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EXECUTIVE  SUMMARY 


The  following  recommendation,  made  in  the  June  1996  briefing  to  the  Commander,  Air  Transport 
Group  (ATG),  served  as  the  impetus  to  the  work. 

•  Raise  the  overall  level  of  systems  knowledge  among  ATG  aircrew  by  developing  teaching  aids 
and  courses  to  accelerate  the  acquisition  of  knowledge  and  to  compensate  for  lowered  fleet 
experience  levels  through  advances  in  training  program  delivery. 

In  this  Volume,  systems  knowledge  training  is  first  placed  within  a  general  risk  management 
scheme  as  an  overall  causal  framework  for  securing  safer  and  more  effective  operations  within  the 
CC-130  community  of  1  Canadian  Air  Division  (1  CAD).  Systems  knowledge  acquisition  is  seen 
to  be  a  latent  risk  factor,  operating  both  as  a  precondition/psychological  precursor  of  risk  at  the 
level  of  the  individual,  and  as  a  systemic  line  management  deficiency  at  the  level  of  the 
organisation.  Formal  evaluation  of  the  state  of  systems  knowledge  fleet-wide  is  needed  to  provide 
feedback  to  the  organisation  in  its  attempts  to  manage  latent  risks. 

Current  systems  knowledge  training  practices  within  the  organisation  axe  reviewed  next,  including 
past  assessments  of  the  need  for  modernisation  of  training.  It  is  concluded  that  while  current 
training  practices  are  sound,  thorough,  and  professional  in  execution,  there  remains  room  for 
improvement  because  downsizing  and  experience  loss  have  impeded  the  ability  of  8  Wing, 
Trenton,  to  achieve  training  modernisation . 

A  selected  review  of  best  training  concepts  and  practices  is  presented,  based  on  the  recent  literature 
on  education  and  training.  Those  concepts  best  suited  to  the  CC-130  training  environment  are 
identified.  The  key  areas  covered  include: 

•  the  importance  of  life-long  learning; 

•  a  just-in-time  knowledge-to-skill  training  model; 

•  problem-based  (i.e.,  case-based),  self-directed  learning; 

•  effective  use  of  interactive  multi-media  for  training  at  the  Operational  Training  Unit  (OTU),  for 
on-the-job  training  (OJT)  and  self-study; 

•  methods  for  the  development  of  superior  mental  models  of  aircraft  systems  and  their 
operational  interdependencies;  and 

•  the  need  for  evaluation  at  three  levels:  trainee  self-evaluation,  OTU/OJT  formal  student 
evaluation,  and  organisation-wide  monitoring  and  evaluation. 

A  set  of  recommendations  is  then  presented,  for  value-added  training  system  improvements  in  the 
area  of  aircraft  systems  knowledge  acquisition.  These  recommendations  are  based  on  the  best 
adaptable  training  practices  that  can  be  delivered  within  current  organisational  restraints,  including: 

•  distributing  more  formal,  detailed  skill  training  to  the  Squadrons,  while  keeping  conceptual 
mental  model  building  the  focus  of  the  OTU; 

•  teaching  the  Awareness  Implications  and  Planning  (ALP)  model  as  a  conceptual  approach  at  an 
early  stage  and  throughout  training  so  that  it  becomes  habitual; 


•  using  more  case-based  teaching,  supported  by  video  simulations  of  good  as  well  as  poor 
systems  handling,  making  sure  that  aircraft  system  interdependencies,  and  the  need  for 
implicational  thinking  and  pre-planning  to  deal  with  these,  are  illustrated; 

•  forging  partnerships  in  the  world-wide  CC-130  community  to  build  a  library  of  cases  quickly; 

•  leveraging  the  Smart  Board  software  pieces  —  deploy  the  Smart  Boards  at  both  the  OTU  and 
OJT  within  the  context  of  both  problem-based  learning  and  self-study,  combining  them  with  a 
case-based  method,  and  intelligent  tutoring  for  self-directed,  continuous  performance 
improvement  —  build  the  tutor  to  provide  feedback  that  will  develop  pilots’  abilities  for 
objective,  independent  self-evaluation  of  the  state  of  their  systems’  knowledge  (i.e.,  meta¬ 
knowledge)  and  motivate  self-initiated  learning; 

•  develop  appropriate  standards  and  evaluation  tools  for  this  type  of  training; 

•  conduct  formal  evaluation  studies  to  document  the  efficacy  of  these  value-added  improvements; 
and 

•  create  a  non-jeopardy  database  to  monitor  and  track  fleet-wide  the  development  of  superior 
systems  knowledge. 

The  Volume  concludes  with  a  summary  of  the  deliverables,  the  implications  for  implementation 

that  are  discussed  throughout,  and  the  resource  requirements  implied  by  adoption  of  these 

recommendations. 
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INTRODUCTION 


This  is  Volume  4  of  a  six  volume  series  describing  the  outcome  of  a  human  factors  study  into  CC- 
130  operations  (see  References  [1-5]  for  details  of  the  remaining  documents  in  this  series).  In  this 
Volume  the  issue  of  needed  improvements  to  the  training  of  systems  knowledge  is  addressed. 
After  reviewing  the  reasons  why  this  was  identified  as  an  area  needing  improvement,  such  training 
is  placed  within  an  overall  risk  management  framework  for  securing  safer  and  more  effective 
operations  within  the  CC-130  community.  Current  systems  knowledge  training  practices  within 
the  organisation  are  then  reviewed,  including  past  assessments  of  the  need  for  modernisation  of 
training.  Following  this,  a  selection  of  best  training  and  education  concepts  is  reported,  based  on 
the  recent  literature  on  education  and  training,  showing  which  of  these  concepts  are  best  suited  to 
the  CC-130  training  environment.  This  leads  naturally  to  a  set  of  recommendations  for  value- 
added  training  system  improvements  in  the  area  of  aircraft  systems  knowledge  acquisition.  It  is 
argued  that  there  is  a  need  to  quantify,  with  suitable  empirical  documentation,  the  magnitude  of 
these  training  improvements  after  deployment,  and  to  monitor,  fleet-wide,  the  evolving  norms 
regarding  pilots’  systems  knowledge.  This  would  provide  feedback  to  the  organisation  in  its 
attempts  to  manage  latent  risks.  Implications  for  implementation  of  these  ideas  for  both  the 
Operational  Training  Unit  (OTU)  and  for  On-The-Job  Training  (OJT)  are  drawn  out  and  reported 
as  to  complete  the  analysis.  Finally,  the  resource  requirements  implied  by  such  changes  are 
enumerated. 

The  following  recommendation  made  in  the  June  1996  briefing  to  the  Commander,  Air  Transport 
Group  (ATG),  served  as  the  impetus  to  the  work. 

•  Raise  the  overall  level  of  systems  knowledge  among  ATG  aircrew  by  developing  teaching  aids 
and  courses  to  accelerate  the  acquisition  of  knowledge  and  to  compensate  for  lowered  fleet 
experience  levels  through  advances  in  training  program  delivery. 

This  recommendation  resulted  from  the  observation  of  inadequate  in-depth  aircraft  systems 
knowledge  demonstrated  by  several  pilots  in  the  Crew  Behaviour  Assessment  Group  (CBAG) 
behavioural  measurement  validation  study  [6].  These  pilots  had  difficulty  recognising, 
diagnosing,  or  ameliorating  some  of  the  aircraft  systems  failures  incorporated  into  this  event-based 
scenario.  While  a  few  limited  observations  do  not  make  a  fleet-wide  trend,  these  observations 
were  embraced  by  ATG  as  perhaps  typical  of  a  more  general  problem  —  at  least  on  an  intuitive  and 
anecdotal  level  since  adequate  measures  of  systems  knowledge  proficiency  do  not  currently  exist 
that  would  document  fleet-wide  trends  in  this  type  of  expertise. 
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SYSTEMS  KNOWLEDGE  IN  THE  MANAGEMENT  OF  RISK 


Within  Reason’s  [7]  systems  approach  to  risk  management,  adopted  as  a  framework  for  the 
activities  of  the  CC-130  study  team  (see  Volume  1  of  this  series  for  a  more  detailed  discussion), 
the  acquisition  of  aircraft  systems  knowledge  can  be  seen  as  occurring  relatively  early  in  the  chain 
of  organisation-wide  activities  in  which  1  Canadian  Air  Division  (1  CAD)  engages.  A  lack  of 
adequate  and  appropriate  knowledge  of  aircraft  systems  can  therefore  be  conceptualised  most 
generally  as  a  latent,  as  opposed  to  an  active,  failure  in  the  accident  sequence.  Such  lack  of 
knowledge  can  be  classed  as  occurring  within  two  of  the  risk  management  system  states  earlier 
defined,  namely  ‘Preconditions’  and  ‘Line  Management  Deficiencies’.  These  two  states  correspond 
to  control  loops  3  and  4,  respectively,  of  Volume  1,  Figure  4  (repeated  below,  as  Figure  1,  for 
convenience). 


FAILURE  TYPES 


FAILURE  TOKENS 


Figure  1.  Feedback  loops  and  indicators  for  the  management  of  system  safety 
(after  Ref.  [7],  Figure  7.9). 
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As  a  Precondition/Psychological  Precursor  of  Risk 

Inadequate  systems  knowledge,  possibly  caused  through  a  variety  of  factors  such  as  experience 
loss,  or  an  erosion  of  training  or  standards,  sets  a  precondition  for  failure  because  key  cockpit 
decision  makers  (Aircraft  Commanders  or  ACs)  are  inadequately  skilled  to  meet,  with  sufficient 
alacrity,  common  aircraft  systems  challenges,  outages  and  failures.  Unsafe  acts  become  more 
probable  in  a  context  of  lowered  or  inadequate  systems  knowledge,  and  the  more  unsafe  acts 
attributable  to  this  source,  the  more  likely  this  latent  failure  may  eventuate  in  accidents  when  1 
CAD’s  defences-in-depth  are  repeatedly  tested  due  to  this  cause. 

It  is  obvious  to  aircrew  (anecdotal  evidence;  see  also  the  study  of  dimensions  of  captaincy  in 
Volume  5  of  this  report)  that  a  key  dimension  of  leadership  and  captaincy  in  terms  of  both  safety 
and  production,  is  a  high  degree  of  systems  knowledge  proficiency.  No  matter  how  it  is  initially 
imparted,  or  later  developed,  embellished,  and  translated  into  skill,  systems  knowledge  is  a 
necessary  pre-condition  for  the  prevention  of  unsafe  acts,  and  for  the  recovery  from  them  as  well. 
In  the  terms  used  previously  [6,  8],  pilots  must  have  a  well-articulated  and  organised  mental  model 
of  each  aircraft  system  and  of  each  system’s  interactions  with  other  systems. 

These  mental  models  are  complex  cognitive  representations  of  the  various  systems’  functions. 
Such  cognitive  representations  allow  heightened  abilities  for: 

•  perception  and  situational  awareness; 

•  judgement; 

•  implicational/inferential  thinking; 

•  pre-planning;  and 

•  dynamic  decision  making. 

Each  of  these  functions  of  aircraft  systems  mental  models  allows  for  faster  and  more  accurate 
assessment  and  action,  buying  valuable  time  in  the  Information  Processing  (IP)  model1  trade-off 
between  the  time  required  to  do  what  must  be  done  and  that  time  available  to  do  it  [9].  In  addition, 
superior  knowledge  representation  of  systems  implications  allows  for  preplanning  and  decision 
making  that  can  ‘buy  time’  when  necessary,  again  insuring  that  the  IP  Model  ratio  does  not  exceed 
one.  Under  such  conditions  of  superior  aircraft  systems  proficiency,  advanced  systems 
knowledge  representations  become  skill-like  in  their  implementation.  This  is  a  simple  description 
of  the  preparedness  that  comes  with  high  skill  levels;  well-developed  mental  models  about  aircraft 
systems  contribute  to  more  effective  and  efficient  information  processing  about  those  systems,  as 
well  as  to  a  reduction  in  workload  stress  and  an  improvement  in  both  safe  and  efficient  flight 
performance  in  real-time  operations.  Simply  put,  this  state  of  mental  model  functioning  can  be 
taken  as  a  definition  of  expertise  with  aircraft  systems. 


As  a  Line  Management  Deficiency  Risk 

The  development  of  superior  systems  knowledge  that  will  ensure  both  safe  and  efficient  flight 
performance  can  be,  and  is,  imparted  at  many  levels,  and  at  many  time  points  in  the  overall  training 
of  CC-130  pilots.  One  obvious  place  for  intervention,  as  well  as  for  monitoring  systems 
knowledge  to  provide  feedback  to  the  risk  control  system,  is  at  426  Squadron  —  the  OTU.  In  the 
context  of  lowered  fleet-wide  experience  levels  (see  Volume  1  of  this  report),  more  effective  and 
efficient  delivery  of  systems  knowledge  training  can  serve  to  compensate,  at  least  in  some 


1  The  IP  model  was  one  of  the  theoretical  positions  that  guided  this  study  (see  this  report.  Volumes  1  and  5).  The 
IP  model  claims  that  performance,  error  production  and  perceptions  of  workload  are  all  driven  by  the  time  pressure 
(i.e.,  the  ratio  of  time  required  to  time  available). 
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measure,  for  a  loss  of  expertise.  It  was  a  goal  of  the  CC-130  study  team  to  identify  those  teaching 
philosophies,  approaches,  methods,  materials,  delivery  systems  etc.,  that  can  best  add  value  to  an 
already  redoubtable  training  effort  at  426.  In  leveraging  even  further  the  many  solid  approaches  to 
training  as  well  as  training  technologies  currently  in  place  at  the  OTU,  it  is  hoped  to  be  able  to 
recommend  improvements  to  training  delivery  that  will  push  trainees  more  quickly  up  the  learning 
curve  toward  high  proficiency.  It  is  also  hoped  to  be  able  to  document  the  efficacy  of  these 
improvements  with  suitable  empirical  evidence  regarding  their  effects,  in  the  form  of  formal 
evaluation  studies.  In  addition  to  demonstrating  the  efficacy  of  the  proposed  improvements,  these 
studies  will  serve  as  a  model  for  the  kind  of  evaluation  studies  needed  by  the  organisation  to  track 
the  state  of  systems  knowledge  ,  as  pilots  exit  the  various  OTU  courses. 

In  recognising  the  limited  exposure  that  pilots  have  to  426  Squadron,  components  of  training 
outside  the  OTU,  that  are  in  need  of  amplification  or  introduction,  are  also  foreseen.  In  particular, 
OJT  at  Squadron  is  also  a  focus  of  the  study  team’s  recommendations.  The  distribution  of 
superior  training  techniques  to  all  levels  and  kinds  of  training  exposure  is  the  best  way,  it  is 
considered,  to  achieve  a  fleet-wide  improvement  in  systems  knowledge.  As  in  the  case  of  the 
flight  school,  adequate  evaluation,  standards,  and  monitoring  must  be  put  in  place  for  OJT  that  will 
allow  multiple-source  feedback  on  the  normative  state  of  system  knowledge  performance  fleet¬ 
wide.  This  feedback  will  inform  the  risk  management  system  at  a  deep  level  (corresponding  to 
control  loop  4  in  Figure  1)  of  potential,  or  latent,  failure  in  the  area  of  knowledge  of  aircraft 
systems.  Such  a  monitoring  system  is  particularly  important  and  timely  given  the  near-term 
upgrade  to  the  semi-glass  cockpit  for  the  CC-130,  because  past  research  has  shown  that  systems 
interdependencies  in  such  aircraft  exacerbate  the  difficulty  of  training  [10]. 
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REVIEW  OF  CURRENT  TRAINING 


In  Phase  I  of  the  CC-130  study,  anecdotal  evidence  of  inadequate  aircraft  systems  knowledge 
among  some  crew  participants  from  the  crew  performance  validation  study  was  observed,  noted 
and  reported  to  ATG  [6].  In  Phase  n,  activities  were  directed  toward  identifying  and 
recommending  mechanisms  for  the  accelerated  acquisition  of  systems  knowledge  both  within 
formal  and  on-the-job  training.  Because  it  was  Aircraft  Commanders  (ACs)  who  exhibited  the 
clearest  instances  of  lack  of  systems  knowledge  in  the  CBAG  study  (though  it  is  recognised  that  it 
is  likely  only  a  very  small  minority  of  ACs  that  is  not  currently  meeting  appropriate  standards),  that 
focus  was  on  the  acquisition  of  systems  knowledge  among  ACs  at  various  levels  of  training.  The 
current  Phase  II  activities  were  therefore  directed  toward: 

•  conducting  a  review  of  current  426  training  system  methods,  materials,  and  practices; 

•  conducting  a  selected  review  of  the  education  and  training  literature  —  both  theory  and  current 
practices;  and 

•  making  recommendations  for  value-added  improvements  to  the  training  system  in  terms  of 
effectiveness,  efficiency,  and  evaluation. 


Review  of  426  Squadron  Training 

The  review  of  426  Squadron  training  philosophy  and  regimes  included  interviews  with  trainers  and 
course  developers,  reviews  of  written  course  materials  and  limited  attendance  at  some  courses. 
The  focus  was  on  conversion  training  and  the  upgrade  course,  since  it  was  pilots’  (particularly 
ACs’)  system  knowledge  more  than  systems  knowledge  at  any  other  crew  position  that  most 
directly  affected  the  outcomes  of  the  flights  we  observed  in  the  validation  study. 

This  review  confirmed  that  some  excellent  training  resources  exist  at  the  Squadron  currently  for  the 
acquisition  of  systems  knowledge,  and  that  a  starting  infrastructure  exists  to  expand  upon  these 
already-excellent  offerings.  It  is  considered  that  current  426  Squadron  training  is  well-grounded  in 
education/training  theory,  is  both  comprehensive  and  thorough  in  coverage,  and  is  professional  in 
execution.  Future  efforts  at  adding  value  to  this  system  should  use  these  existing  elements  as  a 
base  for  what  would  be  characterised  as  a  small  to  moderate  re-engineering  of  current  instruction. 
However,  there  are  areas  that  can  be  leveraged  further,  some  that  would  be  classed  in  the  area  of 
training  effectiveness,  and  some  in  the  area  of  training  efficiency. 

Particular  strengths  observed  in  the  training  of  systems  knowledge  include: 

•  clear,  lesson-specific  instructional  goals; 

•  comprehensive  course  materials; 

•  effective  examples  and  scenarios; 

•  use  of  a  multi-tiered  approach  (i.e.,  lectures,  mock-ups,  part-task  simulations,  full-motion 
simulations,  etc.); 

•  effective  use  of  cases; 

•  presence  of  experts  in  the  classroom  to  address  problems/questions;  and 
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•  use  of  smart  board  technology  as  the  mechanism  of  instructional  delivery  in  certain  instances 
(i.e.,  fuel,  and  hydraulics  at  present). 

Areas  of  potential  improvement  at  426  Squadron,  and  for  OJT,  that  were  identified  include: 

•  striking  a  new  relative  balance  between  the  conceptual  content  and  the  detail  communicated 
about  systems; 

•  use  of  OJT/Squadron  Standards  Officers  to  continue  the  conversion  of  knowledge  to  skill  in  a 
way  consistent  with  and  co-ordinated  by  trainers; 

•  an  operations-specific  training  needs  assessment  conducted  at  the  various  Squadrons  to  tailor 
some  training  toward  customers  differing  in  role  requirements; 

•  updating  materials  and  examples; 

•  use  of  even  more  facilitation-type  teaching  with  case  study  examples; 

•  explicit  methods  for  developing  superior  mental  models  of  aircraft  systems  that  include 
awareness,  implicational  thinking,  planning,  and  decision  making  regarding  those  systems; 

•  explicit  methods  for  developing  mental  models  of  aircraft  system  interactions; 

•  opportunities  for  training,  developing  and  exhibiting  systems  knowledge  and  systems 
knowledge-relevant  decision  making  in  dynamic,  more  realistic,  event-based  scenarios  (low-  to 
high-fidelity); 

•  completion  of  the  outstanding  smart  boards  for  systems  not  yet  represented  and  taught  in  this 
mode; 

•  increased  functionality  of  the  smart  boards  (e.g.,  to  allow  and  stimulate  implicational  thinking 
about  systems); 

•  improved,  formal  evaluation  and  organisational  feedback  at  every  stage  of  training;  and 

•  provision  of  tools  and  motivation  for  career-long  self-study,  student  feedback  and  the 
development  of  self-evaluation  skills. 

The  review  of  systems  knowledge  training  was  done  in  conjunction  with  a  review  of  current 
education  and  training  practices.  The  latter  served  as  a  filter  through  which  to  view  current  CC- 
130  training  practices.  Before  turning  to  this  review,  it  is  instructive  to  compare  the  CC-130  study 
group’s  independent  assessment  of  systems  knowledge  training  circa  1997,  with  one  conducted  by 
ATG  in  1990.  In  some  respects  the  review  and  recommendations  made  some  7  years  ago  strongly 
foreshadow  the  position  taken  in  this  Volume. 


Hypher’s  Study  of  CC-130  Pilot  Training  Modernisation 

In  1983,  Mr.  Peter  Hypher,  then  Operational  Research  Advisor,  reviewed  the  CC-130  training 
systems  at  the  Operational  Training  Unit  as  part  of  the  Commander  ATG’s  initiative  to  modernise 
the  pilot  and  aircrew  training  systems  [11].  He  updated  his  review  in  1990,  under  contract  to  the 
Operational  Research  and  Analysis  Establishment  [12].  Both  reviews  distinguished  between  the 
concepts  of  cognitive  and  psycho-motor  skills  acquisition,  emphasised  the  need  for  a  progressive 
learning  approach,  and  sought  to  encourage  the  adoption  of  progressive  training  aids. 

Hypher’s  main  arguments  were: 

•  that  learning  of  cognitive  and  psychomotor  skills  within  ATG  took  place  largely  in  isolation 
and  needed  to  be  integrated/altemated  all  along  the  continuum  from  the  OTU  Conversion,  to 
OJT,  to  Upgrade;  and 

•  that  the  ideal  training  system  would  incorporate  training  aids  ranging  along  a  continuum  from 
textbooks  and  lectures,  to  computer-based  training  (CBT)  lessons,  to  part-task  trainers  (PTT), 
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to  cockpit  procedures  trainers  (CPT),  to  the  full  motion  simulator  (Sim),  and  finally  to  the 
aircraft. 

By  1990,  he  saw  CBT  as  desirable  and  feasible  and  spent  much  of  his  report  reviewing  and 
recommending  it  as  the  best  method  for  transmitting  basic  knowledge  and  training. 

Since  1990,  several  aids  have  been  added  to  the  training  system,  including  the  cockpit  mock-up 
(CMU)  arising  out  of  the  CC-130  Avionics  Update  Programme  (AUP),  and  the  Smart  Board 
systems  packages  (fuel  and  hydraulics  delivered,  with  remaining  systems  on  order:  electrics, 
propellers,  etc.),  that  go  a  long  way  toward  meeting  the  recommendations  Hypher  made. 
Importantly,  many  of  Hypher’ s  observations  with  regard  to  the  needs  of  the  training  system  hold 
as  true  today  as  they  did  7  years  ago.  Organisational  stresses  in  the  form  of  downsizing  and  fleet¬ 
wide  pilot  experience  loss  have  prevented  426  Squadron’s  ability  to  modernise  its  training  as 
rapidly  as  hoped. 

When  Hypher’ s  arguments  are  applied  to  the  issue  of  systems  knowledge  acquisition,  many  of  the 
assertions  and  recommendations  are  accepted,  but  issue  is  taken  with  several  points  too.  Without 
exhaustively  reviewing  his  main  contentions  (which  stand  in  his  own  companion  document),  the 
areas  of  basic  agreement  with  his  training  system  vision,  and  areas  where  there  are  differences,  are 
summarised  below. 

It  is  agreed  with  Hypher  [12]: 

•  that  the  present  OTU  systems  knowledge  treatment  "...over-teaches  the  technical  detail..." 
(P-17); 

•  that  there  needs  to  be  more  emphasis  on  "...good  airmanship  and  airspace  awareness  initially 
than  on  technical  detail..."  (p.  22)  at  the  OTU; 

•  that  the  OTU  tends  to  teach  "...ten  years  of  systems  failures..."  (p.  3)  in  a  few  days  at  the 
Conversion  course  and  that  this  overloads  the  student; 

•  that  there  is  a  need  for  a  progressive  learning  approach  in  the  area  of  systems  knowledge 
acquisition  (p.  3); 

•  that  there  is  a  need  for  progressive  training  aids  to  teach  systems  knowledge  (p.  19); 

•  that  the  present  training  system  consists  (still)  of  too  much  chalk-and-talk  with  regard  to 
systems  knowledge  (p.  7;  though  this  is  changing  with  the  introduction  of  each  new  Smart 
Board  module);  and 

•  that  a  rating  system  be  introduced  to  assess  the  training  value  for  each  aircraft  system  of  each 
operational  training  flight  (p.  77). 

His  call  for  the  introduction  of  self-paced  study  is  also  accepted,  though  there  are  differences  on 
the  best  mechanism  for  this.  Hypher  recommended  traditional  CBT,  which  was  state  of  the  art  at 
the  time,  and  this  represents  the  main  area  of  apparent  discrepancy  between  the  recommendations 
of  the  CC-130  study  team  and  those  of  Mr.  Hypher.  While  it  is  true  that  traditional  CBT  has  many 
advantages  over  lectures,  including  shortening  systems  knowledge  acquisition  times  and  bolstering 
knowledge  retention,  the  developmental  efforts  and  costs  attached,  estimated  as  taking  anywhere 
between  250  to  800  person  hours  of  preparation  per  hour  of  CBT  instruction,  and  as  costing 
between  $15,000  to  $30,000  per  hour  of  final  CBT  product,  are  prohibitive.  Moreover,  these 
sorts  of  estimates  are  for  classical  CBT,  which  amounts  in  many  cases  to  expensive  page-turning 
software  of  largely  text-based  material.  To  fully  realise  the  advantages  of  sound,  video,  animation, 
and  simulations  within  a  classical  CBT  environment  would  add  incrementally  to  these  costs. 

ATG  has  not  in  the  past  chosen  the  classical  CBT  option  for  training  systems  knowledge,  and  in 
hindsight  the  study  team  concurs.  CBT  is  not  the  appropriate  alternative  even  at  this  time, 
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especially  in  light  of  the  development  of  another  (but  relatively  less)  cost-intensive  vehicle  for 
instruction  —  the  Smart  Board  CDs,  These  can  powerfully  substitute  for  CBT  —  indeed,  they  are 
a  form  of  it  —  while  at  the  same  time,  it  is  believed,  achieve  considerable  training  efficacy  (it  is 
argued  below  that  this  training  efficacy  should  be  documented  empirically  via  formal  evaluation 
research  using  objective  performance  measures,  evidence  that  will  quantify  the  Smart  Boards’ 
effectiveness  and  increase  their  marketability  in  other  C-130  communities). 

Recent  advances  in  authoring  systems  for  multi-media  training  suggest  that  using  commercially 
available  authoring  software  may  now  be  viable  for  the  creation,  maintenance  and  revision  of 
multi-media  courseware  for  the  CC-130.  This  flexibility  in  design  and  the  ability  to  amend  lesson 
plans  has  been,  heretofore,  largely  lacking  among  CBT  packages.  These  can  now  be  usefully  and 
easily  employed  as  courseware  generators  among  multi-media  literate  trainers.  It  is  recommended 
that  caseware  and  courseware  be  built,  placing  as  their  centrepieces,  the  Smart  Board  system 
simulator  module.  Emphasis  on  preplanning  and  decision  making  will  be  manifested  as  lesson 
objectives  that  surround,  and  are  placed  ‘on  top’  of  the  basic  Smart  Board  capability.  Were  Mr. 
Hypher  to  have  been  writing  today,  given  his  stated  recommendation  to  integrate  sound,  video, 
and  simulation  within  classical  CBT,  it  is  suspected  that  he  would  in  agreement  as  to  the  training 
potential  inherent  in  the  Smart  Board  systems. 

Another  key  area  of  disagreement  in  the  learning  model  adapted  for  the  current  review  of  CC-130 
training,  as  compared  with  Hypher’ s  own  recommendations,  is  his  statement  that  conversion 
training  should  emphasise  “...’how  to  operate  systems’  rather  than  ‘how  they  operate’...”  (p. 
129).  It  is  believed  that  a  sound  conceptual  basis  —  understanding  how  and  why  the  system 
operates  as  it  does  —  provides  a  key  knowledge  structure  within  which  technical  detail  and  skill 
can  be  incorporated  and  retained.  Enough  how  to  operational  knowledge  must  be  imparted  to 
allow  aircrew  to  contribute  as  useful  team  members  upon  exiting  the  conversion  course,  but  a 
sound  conceptual  understanding  is  a  necessary  pre-condition  for  differentiated,  integrated,  and 
retrievable  systems  knowledge.  This  is  especially  the  case  as  the  issue  (increasing  in  importance 
with  the  Avionics  Update  Project  [AUP])  of  interconnected  systems  functioning  [13]  is  broached. 
Situational  awareness,  implicational  thinking,  pre-planning  ability,  and  decision  making  are  all 
facilitated  by  a  sound  mental  representation  of  systems’  causal  interactions. 

One  of  Hypher’ s  stated  reasons  for  suggesting  CBT,  was  to  secure  self-study  opportunities  and 
access  for  students.  Digital  compression  and  other  computer  technology  advances  since  the  time  of 
his  writing  now  make  network-hosted,  fully  interactive  multi-media  CBT  with  simulation 
components  realisable  in  the  near  term  as  a  tool  for  systems  knowledge  training.  The  Smart 
Boards  represent  software  pieces  that  can  be  fairly  easily  adapted  to  be  self-paced,  self-directed 
distance  learning  applications,  concurring  with  and  delivering  Hypher’ s  earlier  vision  for  career- 
long  currency  maintenance  and  performance  improvement. 


Review  of  Education  and  Training  Literature 

Several  of  the  areas  for  improvement,  identified  above,  were  inspired  as  a  result  of  a  selected 
review  of  the  current  literature  on  education  and  training.  Those  trends,  themes  and  principles 
judged  most  pertinent  to  CC-130  pilot  training  are  summarised  below  as  six  main  points.  At  each 
point  the  principle  involved  is  reviewed,  what  is  believed  to  be  its  best  application  within  the  CC- 
130  training  environment  is  described,  and  the  implications  that  adoption  or  further  extension  of 
the  training  approach/principle  would  have  are  outlined.  Recommendations  are  summarised  only 
after  describing  all  the  points  in  order  to  avoid  redundancy,  because  many  of  the  themes  are 
strongly  inter-related.  Resources  that  may  prove  useful  in  future  implementation  are  referenced, 
and  listed  in  the  resource  appendix  attached  (Appendix  1). 
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There  are  six  key  themes  involved  in  this  review.  These  range  from  motivational  factors  that  will 
ensure  more  effective  training  and  an  emphasis  on  conceptual  models,  all  the  way  to  specific  ideas 
concerning  appropriate  self-paced  multi-media  tools  that  will  simultaneously  improve  training 
efficiency,  effectiveness,  and  evaluation.  The  six  themes  to  be  described  are. 

1 .  The  Importance  of  Life-Long  Learning. 

2.  A  Just-in-Time  Knowledge-to-Skill  Training  Model. 

3.  Problem-Based  (i.e.,  Case-Based),  Self-Directed  Learning. 

4.  Effective  Use  of  Interactive  Multi-Media  for  OTU/OJT  and  Self-Study. 

5.  Methods  for  the  Development  of  Superior  Mental  Models  of  Aircraft  Systems  and  their 
Operational  Interdependencies. 

6.  The  Need  for  Evaluation  at  Three  Levels:  Trainee  Self-Evaluation,  OTU/OJT  Formal 
Student  Evaluation,  and  Organisation-Wide  Monitoring  and  Evaluation. 


The  Importance  of  Life-Long  Learning 

Life-long  learning  is  a  concept,  of  great  currency  value  of  late,  in  educational  theory  and  practice 
[14,  15].  Educators  and  trainers  have  long  recognised  that  the  pace  of  technological  invention  and 
change  means  that  the  half-life  of  most  knowledge  is  limited.  In  an  age  of  rapid  advancement,  it 
becomes  requisite  that  practitioners  in  every  field  keep  constantly  in  tune  with  the  latest  trends  and 
developments  in  their  areas  of  expertise.  Responsibility  for  the  pursuit  of  continuous  education 
and  performance  improvement  throughout  the  career-  and  life-span  now  vests  with  individuals  as 
well  as  the  training  arms  of  organisations.  Life-long  learning  is  the  way  of  the  future  for  everyone, 
and  its  need  is  already  here. 

The  concept  of  life-long  learning  is  viewed  as  being  essentially  a  motivational  one.  Eventually, 
through  the  development  of  assessment  instruments  that  will  allow  the  organisation  to  carefully 
identify,  screen,  and  select  those  candidate  pilots  in  possession  of  both  a  natural  propensity  toward 
self-improvement  via  self-motivated  study  and  a  natural  heightened  recognition  of  the  need  to  stay 
current,  1  CAD  will  be  able  to  select  individuals  who  come  bearing  the  right,  ‘life-long  learning’, 
stuff.  However,  such  diagnostic  measures  of  personal  motivation  and  selection  are  not  yet  in 
existence,  and  so  steps  that  are  systemic  and  situational  in  nature  must  be  put  in  place  that  will 
motivate  aircrew  toward  self-improvement  at  the  same  time  as  giving  them  the  kind  of  tools  and 
affording  them  the  kind  of  learning  environment  where  motivated  self-study  can  flourish  early  on 
and  be  maintained  throughout  the  career-span.  In  other  words,  1  CAD  must  create,  even  more 
than  it  has  already  largely  successfully  done,  the  means,  motive  and  opportunity  for  life-long 
learning.  "  c 

There  are  larger  issues  of  organisational  structure,  career  paths,  incentive  systems,  organisational 
commitment  and  the  like  that  will  need  to  be  broached  to  ensure  the  success  of  instilling  a 
philosophy  of  life-long  learning  [16].  This  Volume  will  focus  on  tools  for  self-directed, 
continuous  improvement  rather  than  on  these  larger  organisational  issues.  It  is  believed  that  the 
kind  of  balanced  training  model  proposed  (see  next  section)  will  make  training  more  enjoyable, 
(even)  more  respected,  and  more  motivating  than  it  is  currently,  in  keeping  with  the  kind  of  role 
(and  status)  within  the  organisation  that  the  training  system  must  occupy. 

Implications.  Occupational  environments  that  promote  life-long  learning  must  support  such 
learning  with  proper  tools  and  aiding.  Limited  teaching  budgets  suggest  that  continual 
improvement  and  education  is  done  most  cost-effectively  if  the  student  assumes  major 
responsibility  for  their  upgrading  via  self-study.  In  the  area  of  systems  knowledge  acquisition, 
this  suggests  that  appropriately  gauged  self-study  packages  be  available  to  pilots  on  an  on-going 
basis.  This  also  implies  that  experts  be  available  for  consultation  in  support  of  self-study,  and  that 
software,  for  example,  be  available  to  provide  an  appropriate  degree  of  ‘help’  when  needed  in  the 
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absence  of  such  experts.  Help  can  range  from  the  provision  of  indexes  and  glossaries  during  self- 
study  all  the  way  to  intelligent  tutoring  systems  (see  below).  Since  most  of  these  self-study 
materials  are  likely  to  be  computer-based  (i.e.,  software),  this  also  implies  some  investment  in 
pilots’  computer  literacy  early  on  in  their  training  to  maximise  their  self-study  and  life-long  learning 
potentials. 


A  lust -in -Time  Knowledge-to-Skill  Training  Model 

Effective  training  starts  with  the  conveyance  of  knowledge,  by  which  it  is  meant  that  a  conceptual 
understanding  of  the  subject  based  on  theory  and  examples  is  laid  down,  and  through  practice  this 
knowledge  becomes  appropriately  skill-based.  Simple  principles  of  knowledge  acquisition 
suggest  Siat  a  strong  conceptual  grounding,  or  infrastructure,  is  necessary  in  the  knowledge 
acquisition  process.  Later  detailed  understandings  are  best  learned  and  retained  when  they  can  be 
assimilated  to  an  existing  conceptual  ‘scaffolding’  (i.e.,  assimilation  of  knowledge  is  always  easier 
than  accommodation). 

In  the  present  instance,  this  conceptual  scaffolding  can  be  understood  in  terms  of  solid,  cognitive 
schemata,  or  mental  models,  of  aircraft  system  functioning.  These  mental  models  promote  higher 
order  cognitive  processing,  such  as  inference,  reasoning,  trouble-shooting,  etc.,  because  they 
represent  not  just  what  a  system  does,  but  how  and  why.  Once  a  strong  conceptual  mental  model 
is  in  place,  the  assimilation  of  details  in  practice  is  an  efficient  process.  It  is  considered  that  some 
training  benefits  may  accme  if  the  limited  exposures  pilots  have  to  426  Squadron  is  even  more 
concentrated  than  at  present  on  improving  this  conceptual  understanding  (i.e.,  laying  the  basis  for 
superior  cognitive  representations),  while  OJT  continues  to  concentrate  on  the  assimilation  of 
details  and  the  development  of  skill.  Over-teaching  of  detail  at  the  OTU  is  not  a  wise  use  of  limited 
ground  school  time. 

The  latter  point  relates  to  the  important  issue  of  the  timing  of  learning.  Learning  is  maximised 
when  concepts  are  acquired  ‘just-in-time’,  that  is,  when  they  are  actually  needed  in  practice; 
knowledge  that  is  transmitted  but  is  unrehearsed  quickly  atrophies  (as  anyone  who  has  taken  a 
course  but  not  applied  it  for  several  months  will  attest).  ‘Just-in-time’  is  a  term  that  strongly 
implies  skill  transmission  at  the  squadron  operational  level.  Therefore,  careful  consideration  must 
be  given  to  articulating  when  it  is  that  different  kinds  of  knowledge  are  optimally  acquired  within 
the  entire  training  system.  An  over-emphasis  on  detail  to  the  detriment  of  conceptual 
understanding  at  426,  when  this  is  more  appropriately  transmitted  at  Squadron,  will  not  produce 
long-term  retention. 

It  is  important  to  note  that  for  a  conceptual  scaffolding  to  work,  it  must  be  supported  by  tools  that 
allow  strong  conceptual  understanding  to  be  stimulated  at  OTU  and  strong  knowledge-to-skill 
conversion  to  occur  during  OJT.  The  Smart  Boards  provide  just  such  a  ready  training  tool/aid  for 
the  development  of  stronger  conceptual  understanding  in  the  form  of  the  schematic  visual 
representation  of  aircraft  systems  presented  (usually)  in  the  upper  half  of  their  screen  displays. 
The  ability  of  the  student  to  manipulate  this  schematic  interactively,  and  to  receive  feedback  about 
the  results  of  their  control  actions,  promotes  a  strong  conceptual  understanding  of  how  and  why 
the  system  operates  as  it  does.  In  addition,  the  simulation  capabilities  of  the  Smart  Boards,  in 
combination  with  appropriate  cases  that  teach  operational  skill  in  the  relevant  contexts,  and  at  the 
pertinent  times,  allow  the  tool  to  achieve  the  aim  of  knowledge-to-skill  conversion  as  well. 

Implications.  The  division  of  training  labour  suggested  by  this  re-balanced  training  model 
implies  an  increased  reliance  on  skill  development  at  the  Squadrons,  which,  in  turn,  entails  a 
somewhat  higher  level  of  support  for  training  at  the  Squadron  level  than  currently  exists.  This 
support  should  come  in  the  form  of  transmission  of  the  Smart  Board  learning  tools  to  all 
squadrons,  accompanied  by  training  of  squadron  Standards  Officers  in  their  facilitator  role  within 
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the  scheme  of  problem-based  (case-based),  self-directed  learning  (see  next  section).  Central  co¬ 
ordination,  monitoring,  and  evaluation  of  the  entire  training  model,  when  delivered  in  this  way,  is 
required,  and  should  be  based  most  appropriately  at  the  training  Squadron. 


Pr<?blem-Based _ (i.e.t  Case-Based).  Self-Directed  Learning 

One  influential  mechanism  for  fostering  a  stance  on  education  that  can  be  described  as  ‘life-long’, 
at  the  same  time  that  it  imparts  strong  concept  building,  is  the  use  of  problem-based,  self-directed 
learning  [15,  17-19].  Because  there  are  many  ways  to  implement  what  is  deemed  to  be  a 
problem-based’  and  ‘self-directed’  approach  in  education,  care  should  be  taken  in  defining  these 
terms.  When  this  is  done  it  will  be  discovered  that  not  all  elements  of  the  approach  lend 
themselves  to  easy  adoption  in  the  military  training  environment.  However,  carefully  pruned  the 
techniques  of  this  approach  can  be  usefully  applied  both  to  the  OTU  and  to  OJT  at  the  operational 
squadron  levels.  This  approach  is  summarised,  and  an  attempt  is  made  to  show  how  it  can  be 
usefully  adapted  at  the  OTU  and  for  OJT. 

Problem -Based  Learning.  Classically,  problem-based  learning  as  been  described  as  involving 
the  presentation  to  students  of  specific  real  or  simulated  case  studies  in  a  particular  field  that  present 
a  problem  or  challenge  requiring  solution.  It  is  contended  that  via  such  ‘situated  learning’  new 
material  is  better  understood  and  retained,  because  it  is  learned  in  concert  with  environmental  cues 
to  be  encountered  in  the  real  situations  in  which  the  learning  will  be  relevant  [20]. 

pie  cases  axe  meant  to  provide  a  springboard  for  learning,  and,  importantly,  for  learning  about 
learning.  Typically  directive  forms  of  instruction  such  as  the  chalk-and-talk  lecture  are  not  used 
with  this  approach  and  the  instructor’s  role  is  re-defined  as  being  one  of  an  intellectual  shepherd 
mentor  or  facilitator.  Students  work  both  individually  and  in  small  groups,  to  define  actively  the 
issues  and  questions  to  be  addressed,  the  tasks  to  be  done,  and  the  information  to  be  discovered  or 
recruited.  In  this  way  the  method  is  self-directed  in  addition  to  being  problem-based.  After 
discovering  on  an  individual  basis  the  information  to  be  used,  trainees  then  work  together  with  the 
information  they  have  accumulated  to  synthesise,  discuss,  and  integrate  it,  and  to  make  final 
recommendations  about  the  problem  solution. 

Because  problems  are  multi-determined  and  multi-faceted,  the  information  sought  usually  must  be 
extracted  from  a  variety  of  disciplines  and  sources,  and  in  a  variety  of  ways.  It  is  contended  that 
this  is  a  learning  challenge  and  intellectual  environment  that  is  more  representative  of  the  real-world 
problems  and  performances  expected  of  employees  in  almost  every  field  once  they  graduate  from 
training.  And  it  is  also  asserted  that  such  ‘contextual  learning’  (i.e.,  learning  as  needed,  just-in- 
time,  and  in  the  context  where  it  will  later  be  applied)  makes  the  learning  more  enjoyable  and  more 
effective  as  well. 

The  group-based  learning  portion  is  integral  to  the  process,  as  it  is  a  central  ‘method  of  work’,  a 
method  again  often  representative  of  the  real  world  group  working  environments  in  which  trainees 
will  spend  most  of  their  professional  lives  [21,  22].  Thus,  the  knowledge  gained  about  group 
dynamics  m  the  group-centred  part  of  this  inquiiy/leaming  process  also  transfers  to  the  real  work 
environment,  where  people  are  expected  more  often  to  perform  in  groups  than  alone  (especially  in 
highly  technical  work  environments). 

Co-operation  and  teamwork  skills  are  fostered  from  initial  training  onward  when  group-based 
learning  is  integrated  into  the  method  of  instruction,  as  people  become  aware  of  such  things  as: 

•  how  to  manage  differential  ability; 

•  how  to  contribute  both  to  teamwork  and  taskwork; 

•  how  to  participate  in  others’  learning; 
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•  how  to  read  one’s  own  and  others’  emotional  reactions; 

•  how  to  build  trust; 

•  how  to  manage  and  resolve  conflict  constructively; 

•  how  to  give  feedback  to  one  another  usefully  and  sensitively; 

•  how  to  manage  difficult  people;  and 

•  how  to  instil  as  the  key  motive  of  team  life  the  performance  needs  of  getting  the  job  done  safely 
and  effectively,  rather  than  letting  fear  and  competition  drive  human  relations  and  team 
performance  in  either  the  training  or  real  world  environment. 

Self-directed,  problem-based  learning  is  usually  a  new  mode  of  learning  for  new  inductees  and  as 
such,  care  is  taken  at  the  outset  to  provide  training  to  trainees  in  the  method  of  training  itself,  so 
that  students  understand  the  approach  and  their  responsibilities  within  it.  During  this  initial  training 
trainees  are  engaged  in  learning  how  to  learn. 

Trainers  also  often  need  to  be  trained  in  the  mode  of  facilitation  required,  which  is  usually  quite 
different  from  their  accustomed  roles  as  ‘experts  professing’.  They  must  learn  in  explicit  terms 
how  to  help  students  learn  how  to  learn.  Trainers  are  asked  to  set  cases,  encourage  and  guide 
trainees  without  forcing  or  directing,  provide  a  role  model  of  critical  thinking  and  critical  self- 
examination,  and  assist  group  members  in  self-evaluation. 

Trainers  are  also  responsible  for  the  evaluation  of  each  member  of  the  group  and  for  shepherding 
group  processes  where  necessary.  Evaluations  can  be  formal  and  on-going  conducted  at  the 
end  of  each  session  —  or  can  be  performed  more  infrequently  at  mid-  and  end-of-course.  And  of 
course,  objective  performance  standards  still  serve  as  essential  benchmarks  of  knowledge 
proficiency  against  which  hard  performance  evaluations  can  be  measured. 

Everything  in  this  approach  is  directed  toward  making  the  student  responsible  for  their  own 
learning  and  to  set  patterns  and  habits  of  learning  suitable  to  their  needs.  This  method  of  operation 
should  be  employed  continuously  throughout  their  professional  career.  However,  it  is  important  to 
note  that  while  self-directed,  learning  is  neither  self-willed  nor  self-indulgent  in  this  regime.  Clear 
expectations  for  training,  training  standards  and  performance  objectives  are  set  and  met.  While  the 
system  of  learning  delivery  is  intended  to  be  flexibly  adapted  to  individuals’  learning  styles,  what 
is  not  open  for  negotiation  is  the  requirement  to  produce  training  outcomes  of  effective  and  efficient 
subject  matter  knowledge  and  proficiency.  Therefore,  the  approach  is  self-directed  and  student- 
centred  from  the  point  of  view  of  the  training  delivery  mechanism,  but  the  programme  itself  is  not 
self-directed.  Considerable  guidance  and  modelling  of  best-learning  practices  is  one  of  the  key 
elements  that  trainers  must  provide  in  their  re-defined  role,  and  the  course  content  is  theirs  to  set  as 
they  see  fit.  This,  in  summary,  is  the  ‘how’  of  problem-based  learning  as  it  is  currently  conceived 
and  practised. 

Overview  of  Domains  of  Knowledge  in  Problem-Based  Learning.  The  ‘what’,  or 
content,  resulting  from  this  approach  is  usually  divided  into  three  intended  domains  of  impact. 

1.  A  knowledge  base  is  grown:  Knowledge  consists  of  both  facts  and  concepts.  The  problem- 
based  learning  approach  tends  to  emphasise  the  development  of  sound  conceptual 
understanding  and  models  within  which  to  organise  facts  (i.e.,  conceptual  scaffolding),  as  well 
as  the  knowledge  of  how  to  gain  access  to  specific  information  as  required.  This  places  it  in 
keeping  with  the  training  model  proposed  for  the  OTU,  wherein  a  greater  emphasis  on 
conceptual  understanding  is  recommended  than  is  currently  practised. 

2.  The  second  domain  of  learning  is  skill:  In  this  case  this  will  include  not  only  systems 
operational  skill,  procedural  skill,  and  psychomotor  skill,  but  also: 
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•  critical  appraisal  skill,  such  as  system  situation  awareness,  the  application  of  judgement,  a 
questioning  attitude,  the  ability  to  solicit  and  weigh  evidence  (like  the  quality  of  current 
flight  data),  system  diagnostic  and  trouble-shooting  skill,  etc.;  and 

•  learning  skills,  such  as  the  ability  to  self-identify  deficiencies  in  one’s  own  systems 
knowledge,  the  ability  to  correct  deficiencies,  knowledge  of  resources  available  to  correct 
knowledge  deficits,  and  understanding  the  importance  of  constantly  renewing  one’s  base  of 
knowledge  (i.e.,  motivated  remediation). 

3.  Finally,  problem-based  self-directed  learning  behaviours:  This  is  a  third  kind  of  domain 
knowledge  —  namely  the  acquisition,  development  and  maintenance  of  personal  qualities  and 
attitudes  appropriate  to  the  profession’s  roles  and  tasks.  This  knowledge  includes: 

•  team  leadership  and  followership  skills; 

•  mentoring  skills  modelled  from  the  examples  provided  by  the  intellectual  shepherds  —  the 
trainers;  and 

•  a  motivated  attitude  of  life-long  learning  —  continual  self-improvement. 

Thus,  in  total,  the  process  is  intrinsically  oriented  toward  the  kind  of  personal  and  teamwork 
development  required  of  left-seaters  in  their  professional  conduct  inside  the  cockpit  and  outside  of 
it  as  well. 

The  Case  for  Case-Based  Training.  The  self-directed  learning  philosophy  need  not  be 
problem-based.  It  is  important  to  review  from  a  cognitive  and  methodological  perspective,  why 
case-based  (nee:  problem-based)  teaching  methods  are  likely  to  be  effective  in  meeting  the  goal  of 
fostering  expertise  in  an  accelerated  time  frame  within  the  CC-130  training  program.  Here  is  why 
knowledge  is  better  transformed  into  operational  skill  using  cases. 

Formal  conversion  training  is  often  designed  such  that  operators  receive  a  level  of  training 
necessary  to  produce  ‘competent’  performance,  that  is,  a  performance  level  sufficient  to  produce 
safe  system  operation  [23].  At  the  end  of  formal  conversion  training,  pilots  can  probably  be  best 
described  as  ‘trained  novices’  [24]  rather  than  as  anything  approaching  experts.  At  this  point  their 
acquired  knowledge  is  unlikely  to  be  well-differentiated  or  well-integrated,  characteristics 
classically  associated  with  expertise.  On  a  continuum  that  ranges  from  compartmentalised 
knowledge  to  integrated  knowledge,  the  structure  of  the  systems  mental  models  held  by  trainees  at 
this  point  in  their  training  is  likely  to  be  classed  towards  the  fractionated,  or  compartmentalised, 
end  of  the  continuum.  The  training  challenge  presented  by  systems  knowledge  acquisition  is,  of 
course,  to  devise  tools  and  methods  of  training  delivery  that  will  allow  trainees  to  progress  along 
this  continuum  of  integration  toward  the  more  expert  end  —  in  the  absence  of  real  world 
experiences  with  all  likely  or  unlikely  systems  situations. 

With  reference  to  specific  kinds  of  knowledge  to  be  trained,  pilots  will  have  three  sorts  of 
understandings  represented  in  their  developing  mental  models  of  aircraft  systems  at  this  point  in 
their  post-conversion  course: 

•  declarative  knowledge, 

•  procedural  knowledge,  and 

•  operational  skill  [25]. 


Declarative  knowledge  is  the  factual  knowledge  possessed  about  the  system,  things  its 
components,  controls,  and  functions.  Procedural  knowledge  refers  to  the  rule-based  knowledge 
possessed,  such  as  how  to  operate  the  system  and  how  to  perform  sub-component  tasks. 
Operational  skill  is  the  ability  to  integrate  the  first  two  forms  of  knowledge,  the  declarative  and  the 
procedural,  in  a  goal-directed,  multi-task,  time-  and  resource-constrained  environment,  in  other 
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words,  in  the  kind  of  dynamic  decision  making  environment  that  the  CC-130  flightdeck  represents. 
It  also  includes  executive  function  sorts  of  knowledge,  such  as  recognising  when  activities  are 
required  (situational  awareness,  implicational  thinking,  prioritisation,  etc.)  and  knowing  how  to 
co-ordinate  these  activities  to  meet  the  demands  of  real-time  operations  (e.g.,  pre-planning, 
managing  time,  allocating  aircraft  and  team/human  resources,  etc.). 

Expertise  is  most  related  to  the  last  of  these  three  knowledge  contents  —  the  operational  application 
of  knowledge  in  real  time.  Studies  have  demonstrated  that  operators  matched  on  their  declarative 
and  procedural  knowledge  (as  assessed  by  test  scores),  but  differing  in  expertise,  show  marked 
differences  in  their  operational  performance  effectiveness  [26].  Trained  novices  typically  exit  from 
formal  training  with  incomplete  declarative  knowledge  and  incomplete  and  only  loosely  integrated 
procedural  knowledge,  yielding  a  ‘buggy’  cognitive  representation  of  each  system  [27]. 

Usually,  expertise  is  obtained  over  time  —  through  the  accumulation  of  real  world  experiences  that 
are  integrated  into  the  basic  mental  conceptual  model.  By  experiencing  a  wide  range  of  situations, 
acting  to  control  systems  within  those  situations,  and  receiving  feedback  about  the  results  of 
control  actions  attempted,  pilots’  declarative  and  procedural  knowledge  becomes  integrated  into 
operational  skill.  Unfortunately,  OJT  sometimes  does  not  expose  all  pilots  to  all  pertinent 
situations,  so  that  line  flying  accumulated  over  a  period  of  time  does  not  always  build  expertise 
sufficiently.  In  addition,  after-action  or  after-event  reviews  may  be  missed  [28],  insufficiently 
detailed,  or  informal  at  the  Squadron  level,  causing  valuable  training  lessons  to  be  foregone  and 
learning  opportunities  to  be  lost. 

Herein  lies  the  great  potential  of  a  case-based  self-directed  training  system  for  use  primarily  at  OJT. 
By  collecting  and  articulating  in  the  form  of  case-based  learning,  a  widely  representative  sample  of 
typical  actions  and  Standard  Operating  Procedures  (SOPs),  as  well  as  known  incidents,  accidents, 
anomalies,  and  typical  faults,  one  can  with  simulation  and  a  repertoire  of  cases,  approximate  the 
breadth  and  depth  of  training  that  would  normally  occur  over  a  more  extended  period  of  time. 
Encapsulating  the  organisation’s  knowledge  in  this  way  also  serves  to  preserve  its  organisational 
memory  of  ‘war  stories’  and  ‘hangar  flying’  — its  informally  archived  cases. 

Note  that  training  success  will  depend  in  large  measure  upon  the  selection  of  appropriate  training 
scenarios  under  a  case-based  approach.  ‘Appropriate’  entails  a  choice  of  case  scenarios  that 
provides  the  proper  environmental  contexts  and  cues  that  reflect  the  way  systems  knowledge  is 
actually  used  in  that  system’s  domain.  If  chosen  or  built  wisely,  the  simulator  cases  will  promote 
the  integration  of  knowledge  and  enhance  operational  skill  [29].  By  trading  cases  with  other  fleets 
operating  the  C-130  airframe,  an  even  fuller  repertoire  of  situations  can  be  represented. 

Problem-Based  Learning’s  Best  (and  Appropriate)  Practices  for  OTU/OJT.  It  is 
recognised  that  426  Squadron  OTU  currently  employs  a  variant  of  a  problem-based  learning 
component  in  its  use  of  case  studies.  A  radical  re-design  of  the  training  system,  in  favour  of  a 
completely  problem-based  approach,  is  not  suggested.  Chalk-and-talk  lectures  and  other  forms  of 
teaching  should  continue  to  have  their  proper  place  in  the  methods  of  training  or  delivery  used. 

However,  for  the  future,  when  such  training  challenges  as  the  Avionics  Update  Programme  arise, 
it  is  believed  that  the  benefits  of  life-long  learning  and  self-study  via  self-directed  learning  will 
become  more  necessary  as  mechanisms  to  help  aircrew  remain  current  in  an  increasingly  systems 
knowledge-intense  environment.  While  the  role  of  the  training  instructor  remains  important  and 
will  not  be  diminished  in  this  new  scheme  (in  fact,  it  is  enhanced  and  more  appropriately 
deployed),  students  must  be  given  better  resources,  skills,  and  the  motivation  for  self-directed 
learning  and  self-paced  study  as  an  additional  avenue  toward  continuous  performance 
improvement.  Further,  they  must  be  exposed  to  the  kinds  of  group-based  learning  which  mimic 
their  work  environments  on  the  flight  deck.  Once  the  group-based  format  is  acquired  during 
ground  school  at  426,  it  will  be  easily  recapitulated  as  a  ‘method  of  work’  in  the  operational 
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environments  at  the  squadrons.  Moreover,  it  has  great  potential  as  a  distance-based  group  method 
using  suitable  real  time  communication  across  computer  networks,  allowing  teams  to  be  composed 
from  members  at  a  distance  —  such  as  from  different  squadrons. 

Certain,  limited  aspects  of  the  problem-based  learning  approach  described  above,  can  be  translated 
into  training  practices  that  will  have  a  positive  payoff  for  the  Hercules  pilot  community.  In  fact, 
existing  elements  of  the  OTU’s  current  lessons,  structures  and  philosophy  lend  them  naturally  to  a 
problem-based  approach.  Class  sizes,  for  example,  are  already  small,  much  in  keeping  with  the  5- 
6  member  groups  recommended  by  most  problem-based  programs.  Lessons  are  already  designed 
that  focus  on  systems  failures,  a  good  springboard  for  the  problem-based  mode.  And  a 
compendium  of  ‘war  stories’  (although  as  yet  undocumented)  can  also  act  as  excellent  grist  for  the 
problem-solving  mill. 

The  ‘limited  aspects’  of  the  approach,  referred  to  above  as  appropriate  areas  for  adoption,  are 
summarised  in  the  following  list: 

•  it  is  not  recommended  that  the  entire  training  effort  be  converted  to  this  format  —  lectures  are 
still  the  most  efficient  mode  of  rapid  information  transfer  (except  for  CBT)  when  this  is  what  is 
required,  for  example  in  early  learning; 

•  more  study  must  be  given  as  to  which  specific  lessons  are  best  delivered  in  the  problem-based 
mode; 

•  implementation  should  be  phased,  and  slow  to  moderate  in  speed; 

•  proper  training  in  the  method  for  trainees  must  be  supplied  —  such  training  is  essential  in 
teaching  requisite  skills  for  problem-based  learning  and  in  setting  expectations  for  training  and 
self-efficacy  —  the  latter  factors  have  been  shown  to  be  instrumental  in  the  effectiveness  of 
training  [16]; 

•  proper  training  for  trainers  must  be  executed,  enabling  trainers  to  assume  their  new  roles  as 
facilitators,  mentors,  Subject  Matter  Expert  (SME)  models,  resources  for  self-evaluation,  etc.; 
and 

•  objective  evaluations  must  be  developed  (subjective  forms  of  evaluation,  inappropriate  to  the 
military  training  context,  are  often  used  in  applications  of  problem-based  learning). 

At  present,  the  times,  places,  and  choice  of  learning  path  used  by  the  student  are  still  largely  in  the 
hands  of  the  training  system.  The  results  of  problem-based  learning  approaches  indicate  that, 
within  limits  and  with  considerable  guidance  at  times,  trainees’  learning  is  more  effective  when 
they  have  some  control  over  these  factors.  Students  learns  best  when  they  can  identify  their 
learning  needs,  select  learning  methods  that  suit  them  best,  select  the  location  (time  and  place) 
where  learning  is  maximised,  and  learn  to  identify  through  appropriate  unsolicited,  and  later 
requested,  feedback,  whether  or  not  they  are  achieving  their  goals. 

A  suitable  tool  for  self-paced  study  is  needed  that  would  allow  such  flexibility  in  the  learning  of 
systems  knowledge  and  in  the  continual  upgrading  required  to  remain  current  on  the  complex 
systems  contained  in  the  CC-130  at  present  and  in  the  future.  The  basics  of  such  a  self-study  tool 
already  exists  in  the  form  of  the  Smart  Board  system  simulation  interactive  CD  ROMs.  While 
currently  this  tool  is  used  primarily  in  the  classroom  at  the  OTU,  it  can  be  amended  for  use  as  a 
stand-alone  self-directed  learning  resource  made  available  at  the  Squadrons  and  for  use  at  home. 
As  its  hardware  requirements  are  currently  within  the  abilities  of  most  home  PC  systems  on  the 
market  today  (and  in  the  future),  and  as  the  software  upgrades  to  these  packages  needed  to  facilitate 
self-paced  study  do  not  require  additional  PC  system  capabilities,  this  tool  has  excellent  self- 
directed  learning  potential. 
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To  fully  realise  the  learning  potential  of  the  Smart  Board  ‘part  task  simulators’,  several  ways  in 
which  this  technology  can  be  leveraged  are  suggested  (see  next  sections),  for  example,  by 
combining  them  with  video  re-creations  of  documented  incidents/accidents  and  known  problems, 
by  adding  an  Awareness-Implications-Planning  (AIP)  overlay  module  to  begin  to  explicitly  teach 
AIP  principles  as  habits  of  thought,  and  by  adding  some  intelligent  tutoring  capabilities  (see 
below). 

Implications.  To  summarise,  the  addition  of  a  problem-based  learning  focus  in  the  OTU  and 
beyond  carries  with  it  several  implications: 

•  the  requirement  to  train  trainees  in  this  method  of  work  at  OTU  (initially  at  the  conversion 
course  or  earlier  for  pipeliners),  so  that  they  may  continue  this  form  of  education  at  OJT  —  this 
is  critical  to  its  success  in  the  creation  and  effective  management  of  work  teams; 

•  the  requirement  to  train  the  trainers  in  this  method  both  at  the  OTU  and  at  the  operational  units; 

•  the  need  to  create  self-paced  study  tools  (via  software  extensions  of  Smart  Board);  and 

•  the  need  to  create  and  manage  a  compendium  of  training  cases. 


Effective  Use  of  Interactive  Multi-Media  for  OTU/O.TT  and  Self-Study 

Recent  trends  in  technical  education  point  squarely  to  multi-media  packages  as  a  currently  preferred 
major  mechanism  of  course  delivery.  Efficacy  studies  show  the  reduced  knowledge  acquisition 
times  and  enhanced  knowledge  retention  levels  achievable  via  these  packages,  be  they  traditional 
computer  based  training  (CBT)  or  low  to  mid-fidelity  simulations  [30,  31].  Another  chief 
advantage  of  multi-media  is  its  potential  for  self-study  application  outside  the  classroom. 

Specific  ways  in  which  the  Smart  Boards  may  be  leveraged  to  improve  them  even  further  are  now 
reviewed.  Following  this,  a  recent  development  in  intelligent  tutoring  is  described  —  one  that 
employs  cases  as  method  within  a  multi-media  environment,  in  keeping  with  our  emphasis  on 
problem-  or  case-based  teaching  methods. 

Improvements  to  the  Smart  Boards.  Smart  Board  modules  are  excellent  training  aids,  and 
early  anecdotal  evidence  of  their  efficacy  regarding  learning  speed  and  retention  from  426 
Squadron  is  extremely  promising  (personal  correspondence).  It  is  understood  that  their  initial 
development  was  for  the  training  of  technicians,  but  it  is  believed  that  they  can  be  even  further 
enhanced  to  achieve  the  kind  of  pilot  knowledge  and  skill  acquisition  objectives  enumerated  in  this 
Volume.  They  should  therefore  be  finished  on  a  fast  track  for  all  systems  not  currently  represented 
in  this  format,  and  appropriate  applications  should  be  developed  for  the  conversion  course,  pilot 
upgrade  to  left  seat  course,  and  Flight  Engineer’s  (FEs’)  course  at  the  OTU,  as  well  as  for 
applications  at  the  Squadron  level.  In  the  terms  already  discussed  for  effective,  efficient,  and 
evaluatable  training,  the  Smart  Boards  lend  themselves  readily  for  application  because: 

•  the  schematics  provide  a  visual  conceptual  scaffolding  of  the  kind  whose  importance  has  been 
emphasised  as  the  central  learning  objective  for  pilots  at  426  Squadron; 

•  the  Smart  Board  software  is  easy  to  distribute  to  Squadrons,  hardware  infrastructure 
notwithstanding  —  the  full  system  dynamic  representation  in  the  ‘lower’  half  of  the  display  is 
an  effective  detail/skill  developer  that  can  be  resourced  repeatedly  at  the  Squadron  level  to 
enhance  the  conceptual  underpinning  imparted  at  ground  school  and  to  integrate  declarative  and 
procedural  knowledge  into  operational  skill; 
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•  with  relatively  simple  enhancements,  the  AIP  formulation  can  be  integrated  with  current  Smart 
Board  representations  (see  next  section)  —  the  focus  again  will  be  on  system  implications  and 
system  interdependencies;  and 

•  Smart  Board  feedback  functionality  can  be  incorporated  anywhere  along  a  continuum  from 
non-intelligent  interfaces  (e.g.,  pop-up  windows  with  verbal  answers)  to  intelligent  ones  (e.g., 
intelligent  tutors)  —  because  an  eventual  goal  is  to  allow  self-directed  learning,  it  is 
recommended  that  at  least  some  measure  of  intelligence  is  put  behind  the  tutoring  system. 

The  potential  for  self-study  using  the  Smart  Board  is  obvious  (see  last  point).  Some  direction  will 
have  to  be  given  as  to  the  appropriate  (or  standards-based)  level  at  which  the  material  is  expected  to 
be  learned  for  a  given  crew  position,  but  the  potential  for  learning  to  whatever  depth  is  desired  by 
pilots  should  be  preserved.  The  following  improvements  are  recommended. 

•  Mow  aircrew  to  interact  with  systems  simulations:  when  illustrating  cases,  having  a  dynamic 
representation  of  aircraft  systems  in  real  time  in  the  form  of  a  Smart  Board  display,  as  well  as  a 
real-time  video  simulation  of  cockpit  events  (played  on-screen  eventually  as  hardware  is 
upgraded,  and  via  a  parallel  presentation  medium  initially),  will  be  a  powerful  aid  to  training. 
The  instructor  should  be  able  either  to  stipulate  the  end  point  of  the  dynamic  system  or  to 
render  ineffective  certain  student  actions,  so  as  to  be  able  to  simulate  real  aircraft  cases  of 
incidents  and  accidents  where  pilot  control  actions  proved  ineffective. 

•  Mow  monitoring,  evaluation  and  feedback  of  student  actions:  whether  in-course  or  in  self- 

study  mode,  the  smart  boards  can  be  engineered  to  come  equipped  with  ‘canned’  starting 
system  parameters.  By  recording  obvious  indicators  such  as  pilot  control  actions  and  their 
sequences,  reaction  times,  and  responses  to  open-ended  queries  (e.g.,  problem  awareness  and 
diagnosis,  implications,  pre-planning,  knowledge  of  system  interactions,  etc.),  instructors  (or 
an  intelligent  tutor)  can  evaluate  a  student’s  systems  knowledge.  Such  measures  taken  over 
time  would  allow  tracking  of  student  progress.  Used  as  a  formal  evaluation  method,  normative 
data  could  be  collected  about  fleet-wide  trends  using  a  non-jeopardy  data  base.  Especially  in 
self-study  mode,  feedback  to  the  student  can  be  given  in  a  way  that  provides  raw  data  for  the 
development  of  meta-knowledge  (knowledge  about  the  state  of  one’s  knowledge),  an  important 
precursor  of  expertise  and  objective  self-evaluation  about  proficiency  in  knowledge  and  habits 
of  thought.  6 

Intelligent  Tutors  for  Simulation.  It  is  encouraging  to  have  recently  found  a  training  tool 
with  many  of  the  formal  properties  that  have  been  reviewed  above  as  being  desirable  best  training 
practices.  This  tool  was  conceptualised  independently  of  this  review,  but  is  convergent  in  its 
recommended  methods  for  training  delivery.  It  is  reviewed  now  in  some  detail,  noting  at  the  outset 
a  number  of  ways  in  which  it  is  similar  and  number  of  ways  in  which  it  differs  from  the  current 
proposals  for  renewed  training  delivery. 

Chappell  and  Mitchell  [27]  recently  described  an  architecture  for  a  computer-based  training  system 
for  operators  of  complex  dynamic  systems,  the  Case-Based  Intelligent  Tutoring  System  (CB-ITS). 
This  can  serve  as  a  conceptual  roadmap  for  leveraging  the  investments  made  to  date  by  8  Wing  in 
the  Smart  Board  systems  knowledge  delivery  modules.  It  is  similar  to  the  approach  recommended 
in  this  Volume,  as  follows; 

•  it  is  intended  to  help  pilots  enhance  and  maintain  expertise; 

•  it  is  inherently  case-  or  problem-based,  providing  users  with  normal  as  well  as  unusual 
operating  situations; 

•  it  emphasises  the  acquisition  of  sound  conceptual  underpinnings  for  systems  knowledge  —  the 
‘how’  and  ‘why’  of  systems  functioning,  not  just  the  ‘what’; 
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•  it  includes  a  simulator  that  allows  students  to  practice  translating  their  procedural  knowledge 
into  operational  skill; 

•  it  includes  an  intelligent  tutor  with  variable  ability  to  provide  intelligence  and  instructional 
control  as  a  function  of  student  needs;  and 

•  it  is  inherently  closed-loop  and  systems-based  (e.g.,  in  its  use  of  feedback  to  the  student  to 
help  null  the  error  signals  while  learning). 

Chappell  and  Mitchell’s  approach  is  different  to  the  approach  to  instructional  delivery 
recommended  in  this  Volume,  in  the  following  ways: 

•  it  is  designed  primarily  as  a  self-directed  form  of  OJT,  whereas  the  approach  recommended  for 
the  CC-130  would  marry  it  to  both  OTU  and  OJT  in  terms  of  a  common  approach  and  method 
of  work; 

•  it  does  not  integrate  an  AIP  framework  (see  next  section); 

•  it  does  not  integrate  teamwork  into  the  method  of  work; 

•  it  has  less  emphasis  on  developing  high  level  self  monitoring  skills  and  self-evaluation;  and 

•  it  uses  a  different  simulation  platform. 

These  points  stated,  the  utility  of  intelligent  tutoring  systems  (ITSs)  lies  in  their  ability  to  tailor 
instruction  to  the  needs  of  individual  students.  To  do  so,  such  systems  contain  four  components: 

•  a  domain  expert, 

•  a  student  model, 

•  the  instructional  method,  and 

•  the  tutor  interface  [32,  33]. 

The  domain  expert  is  a  model  representing  knowledge  that  characterises  expert  performance, 
including  declarative  and  procedural  knowledge,  and  operational  skill.  The  student  model 
represents  the  tutor’s  changing  knowledge  of  the  student’s  knowledge.  By  comparing  at  any  point 
the  domain  model  with  the  student  model,  the  tutor  selects  teaching  strategies  such  as  cases, 
illustrations,  hints,  coaching,  etc.,  that  will  close  this  expert-student  gap.  Such  contents  are 
delivered  via  interaction  with  the  student  using  the  tutor  interface. 

An  ITS  is  intrinsically  closed  loop  or  cybernetic  in  nature,  in  keeping  with  the  overall  theme  we 
have  struck  regarding  systems  theory.  In  attempting  to  null  the  ‘error  signal’  between  student  and 
expert,  the  ITS  exerts  control  actions  on  the  student.  As  a  student  becomes  more  advanced,  the 
ITS  exerts  less  control  and  models  its  expertise  to  a  decreasing  degree  along  a  continuum  ranging 
from  active  prompting,  to  providing  feedback  when  it  notes  an  error  or  anomaly,  to  minimal 
intervention  (i.e.,  only  providing  updates  on  systems  at  checkpoints,  when  errors  are  no  longer 
recoverable). 

Collins  [20]  suggests  that  an  ITS  delivers  a  form  of  ‘cognitive  apprenticeship’  to  the  student  by 
demonstrating  expert  performance  in  the  context  of  realistic  task  scenarios.  In  doing  so  it  helps 
students  to  better  integrate  knowledge  of  systems  and  procedure  into  operational  skill  because  the 
knowledge  of  systems  required  for  safe  and  efficient  flight  performance  that  is  illustrated  occurs  in 
a  multi-task,  real-time  simulation.  Cognitive  apprenticeship  has  been  used  with  some  success  in 
several  training  systems  that  teach  performance  and  decision  making  in  naturalistic  decision  making 
environments  with  complex  systems  [25,  34]. 

During  simulation,  CB-ITS  allows  students  to  make  the  implicit  knowledge  contained  in  its  expert 
model  explicit  at  any  point  in  time.  The  student  simply  must  ask  why  the  steps  in  an  activity  may 
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have  been  prescribed  by  the  tutor  to  receive  reasons  that  describe  the  pertinent  declarative  and 
procedural  knowledge  and  integrate  it  into  the  situation  in  progress.  This  too  promotes  the 
development  of  operational  skill  and  perhaps  meta-awareness  of  the  state  of  one’s  knowledge  as 
well. 

Other  expert  systems  offer  similar  capabilities,  and  may  even  allow  better  development  of  students’ 
expertise  and  meta-knowledge.  For  example,  the  expert  system  knowledge  elicitation  package 
EXSYS  prompts  students  with  questions  about  the  state  of  their  knowledge  [35].  During  this 
process  students  may  at  any  point  ask  why  the  question  itself  is  being  asked  by  the  system.  A 
tutor  that  describes  why  it  is  making  queries,  in  the  manner  and  order  in  which  it  is,  can  usefully 
model  the  expert’s  thinking  process.  The  ability  to  reason  about  novel  situations  is  a  hallmark  of 
expertise,  and  among  the  most  important  things  to  model  and  learn  along  the  road  to  expert 
knowledge  is  the  sequence  of  appropriate  questions  to  ask.  The  implicit  causal  model  of  systems 
operations  held  by  the  expert  that  allows  reasoning,  a  model  that  one  would  like  to  have  students 
learn,  is  often  well  exposed  by  observing  the  expert’s  process  of  inquiry,  that  is,  the  processes  for 
generating  new  knowledge. 

In  summary,  several  expert  knowledge  representation  software  simulation  packages  exist  that  can 
be  usefully  built  into  the  Smart  Board  system.  When  incorporated  as  intelligent  tutors,  these  have 
the  potential  to  provide  effective  and  efficient  training  via  self-study. 

Implications.  Implied  in  the  use  of  multi-media  packages  is  that  8  Wing  should  formally 
evaluate  their  efficacy  (as  has  been  begun  for  the  Smart  Board  software).  Chief  downsides  of 
multi-media  include,  of  course,  the  initial  development  costs  where  customisation  is  needed. 
Corporations  are  placing  such  materials  on  their  intranets  for  distance  education  purposes  and  self- 
study,  implying  additional  costs  for  IT  positions  to  support  it,  authoring  software  and  instructor 
training,  course  management  software,  etc.  The  integration  of  an  intelligent  tutor  will  require  an 
initial  phase  of  expertise  extraction  and  system  representation  in  the  domain  expert  model  built  for 
each  aircraft  system. 


jPevelppment  of  Superior  Mental  Models  of  Aircraft  Systems 

Pilots  in  the  CBAG  validation  study  fell  down  on  systems  knowledge  in  two  ways:  first,  because 
they  had  trouble  in  the  initial  recognition  or  diagnosis  of  problems  and  solutions,  or  second, 
because  they  failed  to  adequately  take  into  account  and  plan  ahead  for  possible  implications  of  the 
systems  problems  they  encountered.  By  teaching  as  a  mental  heuristic  the  Awareness- 
Implications-Planning  approach  —  namely,  how  to  be  more  aware  of  developing  system  states, 
how  to  create  a  habit  of  thought  to  consider  the  future  implications  of  the  systems  problems 
experienced,  and  how  to  pre-plan  for  various  systems  and  inter-systems  contingencies  —  pilots 
will  come  to  routinise  in  their  thinking  the  very  kind  of  diagnostic  abilities  they  most  need  to  solve 
systems  problems. 

The  longer  term  view  of  this  approach  is  to  state  that  it  is  consistent  with  the  emphasis  placed 
elsewhere  (this  report,  see  Volume  5)  on  the  theme  of  cockpit  decision  making,  especially  on  the 
part  of  ACs.  In  this  case,  AIP  contributes  to  decision  making  about  systems  in  particular,  and 
about  the  entire  mission,  as  systems  problems  are  considered  in  light  of  the  big  picture. 

Implications.  One  way  to  instil  an  AIP  mental  approach  to  systems  considerations  is  to  query 
pilots  in  mid-simulation  about  what  they  are  thinking  about  the  systems  they  are  operating.  This 
would  require  the  capability  for  the  instructor  to  stop  the  simulation  to  make  the  inquiry  (or  allow 
the  pilot  to  record  their  thinking  in  response  to  a  pop-up  window  query).  As  this  might  prove 
disruptive,  an  alternate  tack  is  to  make  the  AIP  inquiry  part  of  all  after-action  and  post-flight 
reviews  (including  part-task  and  simulator  training),  and  consider  making  it  overtly  manifest  via 
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inclusion  in  training  checklists,  say  during  the  take-off  and  approach  briefs.  At  minimum,  a  simple 
extension  of  instructors’  current  case  scenarios  that  would  ask  pilots  about  systems  implications, 
both  in  the  future  and  for  other  systems  as  new  faults  are  encountered,  would  go  far  in  making  this 
higher  order  mental  model  checklist  habitual. 

Appropriate  and  valid  objective  assessment  instruments  and  standards  need  to  be  developed  that 
would  make  such  implicational  thinking  and  pre-planning  activities  subject  to  formal  evaluation. 
Feedback  will  be  needed  to  allow  students  to  develop  an  awareness  of  their  strengths  and 
weaknesses  in  these  higher  order  cognitive  skills  (especially  in  self-study  mode),  and  the  fleet  can 
be  monitored  squadron-wide  to  index  the  development  of  such  safety  skills  in  the  non-jeopardy 
database. 


The  Need  for  Evaluation  at  Three  Levels 

Consistent  with  the  general  approach  and  framework  of  control  systems  theory,  students  also  need 
appropriate  feedback  in  order  to  evaluate  their  progress  and  achieve  learning  (i.e.,  null  the  error 
signals  they  are  receiving  about  whether  their  actions  on  the  environment  are  perceived  to  move 
them  toward  their  goals).  Mechanisms  for  this  are  many.  The  need  for  evaluation  to  occur  on  at 
least  three  distinct  levels  has  been  identified,  namely: 

1.  Trainee  Self-Evaluation 

2.  OTU/OJT  Formal  Student  Evaluation  and  Organisation-Wide  Monitoring  and  Evaluation;  and 

3.  Formal  Efficacy  Study 

Trainee  Self  Evaluation.  The  development  of  meta-awareness  is  a  main  goal  of  self- 
evaluation.  This  refers  to  the  developing  ability  of  a  student  to  think  about  their  own  learning  — 
its  completeness,  strengths  and  weaknesses,  habits  of  thought,  barriers,  etc.  Only  when  a  student 
is  capable  of  diagnosing  what  they  do  not  know,  and  is  motivated  to  correct  this  deficiency,  can 
they  be  said  to  be  a  self-directed  learner.  This  kind  of  meta-awareness  is  a  demonstrated  correlate 
(and  most  likely,  cause)  of  independent  thinking  and  expertise.  It  must  be  explicitly  modelled  as  a 
habit  of  thought  by  trainers  instructed  about  its  importance,  and  themselves  taught  how  to  articulate 
their  tacit  expertise. 

OTU/OJT  Student  Evaluation  and  Organisation- Wide  Monitoring.  In  preparation  for 
documenting  the  efficacy  of  the  suggested  modifications  to  the  training  system  in  general  (e.g.,  use 
of  more  problem-based  teaching  with  cases)  and  the  Smart  Board  modules  in  particular  (use  of 
cases,  self-study,  intelligent  tutor,  etc.),  standards  of  training  will  need  development  and 
evaluations  of  student  performance  against  those  standards  will  have  to  be  performed.  Student 
evaluation  in  quantitative  terms  must  be  part  of  the  revised  training  package,  along  with 
mechanisms  for  student  feedback  about  performance.  Tracking  pilot  performances  in  the  area  of 
systems  knowledge 

•  to  establish  a  baseline,  and  through  time,  at  exit  from  the  various  pilot  training  courses  at 
OTU,  as  well  as 

•  via  the  creation  of  a  no-jeopardy  data  base  build  on  spot-testing  at  OJT  or  appropriately 
systems  knowledge-intensive,  event-based,  full-motion  simulator  scenario  tests  delivered 
during  continuance  training, 

will  serve  as  mechanisms  for  individual  and  fleetwide  evaluation,  respectively,  of  systems 
knowledge  proficiency. 
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Formal  Efficacy  Study.  Finally,  using  the  assessment  instruments  and  standards  developed 
for  proficient  systems  knowledge  for  each  system,  a  formal  study  of  the  efficacy  of  this  training  as 
compared  to  either: 

1.  a  ‘pre-improvement’  baseline,  or 

2.  another  CC- 130  training  environment, 

should  be  conducted.  The  first  option  is  preferable  empirically,  and  because  the  main  mechanisms 
for  the  re-engineered  course  delivery  are  the  case-based  approach  and  the  Smart  Board 
technologies,  these  are  the  main  avenues  of  potential  improvement  and  therefore  should  be  the 
main  locus  of  evaluation.  If  their  efficacy  can  be  demonstrated,  this  will  be  powerful  evidence  for 
the  teaching  methods  recommended.  This  training  product  can  then  be  ‘sold’  to  other  CC-130 
fleets  world-wide  and  to  the  CF  fleet-wide. 

Implications.  Again,  if  implemented  at  8  Wing,  self-evaluation  training  would  require  feedback 
to  students  and  the  attendant  support  for  it,  especially  via  426  and  Squadron  training/standards 
experts  as  well  as  by  appropriate  software-based  feedback.  New  standards  will  need  to  be 
developed  for  evaluation  purposes  (e.g.,  for  AIP),  and  a  fleet  management  system  and  no-jeopardy 
database  to  accumulate  fleet  wide  information  to  track  trends  in  systems  knowledge  community 
wide  should  be  instituted.  A  formal  efficacy  study  may  require  the  co-operation  of  another  C-130 
fleet  for  comparative  purposes. 
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RECOMMENDATIONS 


In  light  of  the  best  educational  and  training  practices  as  enumerated  in  the  six  themes  in  the 
previous  Section,  and  building  on  the  considerable  strengths  of  the  current  training  system,  the 
following  recommendations  for  systems  knowledge  training  improvements  for  Phase  HI  of  this 
project  are  made.  These  have  been  made  in  light  of  the  constraints  of  staffing  complements,  fiscal 
restraint,  organisational  restructuring,  and  existing  computer  hardware  and  software 
infrastructures,  and  reflect,  it  is  believed,  realistic  goals. 

1*  Distribute  more  formal  training  to  the  Squadrons.  Strong  conceptual  foundations  in  aircraft 
systems  should  be  an  increased  focus  of  training  to  compensate  for  experience  loss,  and  are 
most  appropriately  conveyed  during  the  limited  exposures  pilots  get  to  426  Squadron  (rather 
than  an  over-emphasis  on  detail  in  that  venue).  Convey  more  of  the  detail  at  Squadron,  where 
knowledge  becomes  skill,  and  do  so  more  regularly  and  formally  via  scheduled  trainings 
evaluation,  etc.  The  overall  plan  should  be  orchestrated  by  426,  in  consultation  with 
Squadrons  after  an  assessment  of  their  needs.  An  increased  role  for  OJT  may  imply  more 
training  of  Standards  Officers  in  educational  theory,  use  of  case-based  instruction,  computer 
literacy,  methods  of  evaluation,  etc.  However,  the  results  of  this  process  will  be  that 
Squadrons  will  have  more  of  a  voice  in  creating  the  kind  of  operations-specific  training  of 
systems  knowledge  they  require  on  the  line,  and  will  ‘own’  the  training  to  a  greater  degree. 

2.  Use  more  case-based  teaching,  supported  by  video  simulations  of  good  as  well  as  poor 
systems  handling.  Case-based  teaching  is  both  efficacious  and  enjoyable.  It  can  be  further 
enhanced  in  two  ways.  First,  it  may  be  paired  with  video  simulations  of  real  aircraft 
incidents/accidents,  as  well  as  illustrations  of  best  practices.  The  videos  would  be  created 
using  incident/accident/best  practices  transcripts  as  a  basis  and  using  role-playing  reservists  or 
retirees  as  actors.  This  would  require  use  of  the  full-motion  simulator  for  videotaping  (using 
typically  the  E  and  F  sessions  of  the  training  programme).  Second,  cases  can  be  embedded 
within  the  Smart  Board  presentation  and  learning  format  with  or  without  accompanying  video 
simulations.  One  focus  of  the  case-based  systems  knowledge  teaching  units  would  be  to 
illustrate  system  interdependencies  —  and  the  need  for  implicational  thinking  and  pre-planning 
to  deal  with  these.  Given  the  high  costs  associated  with  linking  the  dynamic  systems  portrayed 
in  the  Smart  Board  modules,  this  is  the  most  effective  and  cost-efficient  approach  to  teaching 
causal  inter-linkage  between  aircraft  systems.  In  order  to  overcome  a  main  obstacle  to  case- 
based  teaching,  the  lack  of  suitably  developed  cases,  it  is  recommended  that  training  partners  in 
the  world- wide  CC-130  community  be  found  that  would  agree  to  share  their  cases. 

3.  Leverage  the  Smart  Board  software  pieces.  Deploy  the  Smart  Boards  at  both  the  OTU  and  OJT 
within  the  context  of  both  problem-based  learning  and  self-study.  Combine  them  with  a  case- 
based  method  of  instruction,  and  intelligent  tutoring  for  self-directed,  continuous  performance 
improvement.  Build  the  tutors  to  provide  feedback  and  develop  pilots’  abilities  for  objective, 
independent  self-evaluation  of  the  state  of  their  systems’  knowledge. 

4-  Teach  the  AIP  model  as  a  conceptual  approach,  at  an  early  stage  and  throughout  training  so  that 
it  becomes  habitual. 
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5.  Conduct  formal  evaluation  studies  to  document  the  efficacy  of  these  value-added 
improvements. 


Recommendations  for  Implementation 

Systems  knowledge  training  will  be  multi-stage,  involving  both  426  and  Squadron-based  training 

and  instruction.  It  should  incorporate: 

•  formal  training  opportunities; 

•  strong  conceptual  mental  model  building; 

•  adequate  practice  opportunities  for  the  development  of  skill  (including  dynamic  decision 
making  with  respect  to  systems  wherever  possible  in  simulations,  part-task  trainers,  etc.); 

•  case-based  learning; 

•  exposure  to  experts; 

•  self-study  training;  and 

•  feedback,  reinforcement,  and  evaluation  for  the  student. 

Details  of  the  implementation  should  be  worked  out  with  8  Wing  and  426  Squadron.  The 

following  tasks  will  need  to  be  considered  to  complete  the  full  implementation: 

•  determining  the  needed  staffing  complement  at  426; 

•  determining  the  staffing  and  support  needed  at  Squadron; 

•  an  organisational  analysis  of  the  co-ordination  needed  between  426  trainers  and  Standards 
Officers  at  Squadron; 

•  a  training  needs  analysis  that  stipulates  the  skills  demonstrable  at  proficiency  for  each  aircraft 
system; 

•  a  training  needs  analysis  that  views  each  system  in  the  context  of  its  operational  environment, 
for  tailored  systems  training  at  Squadron; 

•  an  evaluation  of  how  much  detail  needs  to  be  imparted  at  426  vice  Squadrons; 

•  determining  the  needed  level  of  computer  literacy  for  students  and  trainers,  and  delivering  this 
training; 

•  training  the  trainers/Squadron  officers  in  recent  advances  in  case-based  teaching  theory  and 
their  roles  as  facilitators; 

•  determining  how  to  make  self-study  packages  accessible  to  students  (e.g.,  via  extended  426 
hours,  inter/intra  net  access,  etc.); 

•  completion  of  all  deliverables  (courseware,  caseware,  etc.);  and 

•  a  formal  efficacy  study  of  the  training. 
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CONCLUSIONS 


Despite  downsizing,  and  corporate  raiding  of  its  pilots,  1  CAD,  through  its  OTU  and  OJT, 
continues  to  deliver  a  quality  training  product  in  the  area  of  systems  knowledge  acquisition,  and 
continues  to  provide  innovative  approaches  (e.g.  Smart  Board  Systems)  to  training.  However, 
when  compared  to  other  fleets,  and  reviewed  in  terms  of  the  education  and  training  literature, 
systems  knowledge  training  is  in  need  of  modernisation  in  terms  of  both  effectiveness  and 
efficiency  of  training  philosophy,  materials,  methods,  and  evaluation. 

Modernisation  of  systems  knowledge  training  is  especially  important  in  the  current  context  of 
lowered  experience  levels,  as  a  compensatory  defence  against  latent  failures  that  otherwise  would 
be  caused  by  insufficiently  prepared  pilots  or  inadequately  rigorous  training  regimes  within  the  line 
management  system.  Modernisation  of  systems  knowledge  training  methods  is  also  timely  in  light 
of: 

•  training  advances  made  for  training  semi-glass  and  cockpit  flightdecks; 

•  the  impending  conversion  of  the  CC-130  fleet  to  the  semi-glass  cockpit;  and 

•  the  identification  (both  in  the  aviation  community  in  general,  and  within  the  CF  CC-130 
community  in  particular)  of  training  for  systems  knowledge  complexity  as  a  major 
challenge  for  aircraft  so  equipped. 

Modernised  systems  knowledge  training  will  need  to  be: 

•  multi-staged  and  progressive; 

•  distributed  more  to  OJT  for  improved  integrated  skill  development; 

•  problem-,  or  case-,  based; 

•  retooled  to  add  higher  order  cognitive  functioning  with  regard  to  systems  (e.g., 
systems  situational  awareness,  implicational  thinking  regarding  systems  problems,  and 
planning  for  system  and  inter-system  contingencies); 

•  career-long  via  self-study  opportunities,  which  include  the  development  of  self- 
evaluation  skills  based  on  meta-knowledge  (i.e.,  appraising  what  you  know, 
realistically  knowing  what  you  don’t  know,  and  identifying  what  you  need  to  know  ); 

•  based  on  efficient  training  tools  such  as  interactive  multi-media  simulations 
incorporating  performance  feedback;  and 

•  evaluated  in  the  formal  terms  of  an  efficacy  study. 

Fleet-wide  efficacy  of  the  training  over  time,  as  well  as  long-term  trends  in  absolute  levels  of 
systems  knowledge  fleet-wide,  should  be  monitored  via  a  no-jeopardy  database  of  systems 
knowledge  evaluations  collected  from  several  sources: 

•  from  systems  knowledge  performance  evaluated  on  exit  from  the  Conversion  and  left 
seat  upgrade  course; 

•  from  systems  knowledge  performance  in  appropriately  designed,  event-based  scenarios 
given  during  continuation  training  simulator  sessions  (i.e.,  scenarios  that  incorporate 
systems  knowledge  demands);  and 

•  in  terms  of  reported  incidents  and  accidents  involving  poor  systems  knowledge. 
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Implementation  of  the  systems  knowledge  training  proposed  in  this  Volume  will  require  an 
investment  in: 

•  the  Smart  Boards  System  as  the  comer-stone  training  tool; 

•  leveraging  those  CDs  to  even  greater  educational  benefit  (e.g.,  by  adding  the  AIP  — 
Awareness,  Implications,  and  Pre-planning  —  heuristic,  self-study,  generation  of  a 
library  of  training/simulation  cases,  use  of  simulation  in  concert  with  video  re¬ 
enactments,  development  of  intelligent  tutors  for  self-study,  etc.); 

•  hardware  and  software  infrastructure  improvements  at  OJT; 

•  training  both  trainers  (OTU  and  OJT)  and  trainees  in  the  methods  of  case-based 
instruction; 

•  additional  training  personnel  assigned  to  the  OTU  to  oversee  the  improvements, 
modernisation,  and  direction  of  the  OJT  portion  of  the  training;  and 

•  general  IT  and  computer  training  support. 

Key  benefits  of  the  recommended  systems  knowledge  training  modernisation  are: 

•  increased  safety; 

•  increased  operational  efficiency;  and 

•  a  potentially  saleable  training  package  (i.e.,  philosophy,  structure,  tools,  evaluation, 
etc.)  to  the  C-130  community  world-wide,  and  to  the  CF  fleet-wide. 
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RESOURCES 


The  following  resources  have  been  assembled  from  a  variety  of  sources.  Most  are  brief 
descriptions  culled  from  the  Proceeding  of  the  Society  for  Applied  Learning  Technologies’  (SALT) 
annual  conference  entitled,  Interactive  Multimedia  ‘96,  which  was  held  from  August  21  to  23, 
1996  in  Washington,  DC  and  attended  by  the  author.  A  copy  of  these  proceedings  is  held  at 
DCDEM. 


Lifelong  Learning  Resources 

Contact:  The  Canadian  Alliance  for  Lifelong  Learning  (CALL) 

Suite  400,  20  Bay  Street, 

Toronto,  ON 
M5J  2W1 

Telephone:  (416)  325-421 1 
Fax:  (416)  325-4675 

e-mail:  CALL@gsro.carlton.ca 
Web:  http://gcll.carlton.ca 

CALL  is  a  non-profit  organisation  committed  to  developing  an  action  plan  for  lifelong  learning  in 
Canada  and  around  the  globe.  It  is  associated  with  similar  organisations  around  the  world  through 
its  membership  in  the  World  Initiative  in  Lifelong  Learning,  based  in  Brussels. 

One  of  the  major  foci  of  this  alliance  is  on  practical  issues  involved  in  the  implementation  of 
learner-centred  environments,  methodologies,  technologies,  and  social  strategies.  They  keep  a 
case-study  catalogue  of  actual  examples  of  workplace  models  of  continuous  learning,  geared  to  the 
application  of  the  newest  technology  or  the  newest  learning  models,  such  as  learning 
organisations,  self-directed  learning,  learning  via  the  Internet,  the  electronic  classroom,  distance 
education,  etc. 

Because  the  lifelong  learning  philosophy  and  methods  such  as  problem-based  learning  are  now 
being  applied  down  to  the  kindergarten  level,  they  span  all  age  ranges  in  education.  However, 
there  is  a  particular  focus  within  this  span  on  adult  education  in  general  (e.g.,  within  human 
resources  development  on  productivity  improvement  strategies  based  on  continuous  learning  and 
career  planning)  and  on  workplace  training  in  particular  (e.g.,  creating  and  sustaining  learning 
environments  within  govemment/industry  environments;  impact  of  new  technologies;  electronic 
classrooms;  etc.). 
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Self-Directed  Learning  Resources 

Title:  The  McMaster  Medical  Education  Philosophy 

Contact:  Health  Science  Centre 

Faculty  of  Health  Sciences 
McMaster  University 
1200  Main  Street  West 
Hamilton,  ON 
L8N  3Z5 

Telephone:  (905)  525-9140 

Since  its  inception  in  1969,  McMaster  University’s  Health  Science  Centre  has  pioneered  radical 
innovations  in  medical  education.  With  nearly  30  years  experience  in  the  student-centred, 
problem-based,  self-  and  group-directed  learning  philosophy,  it  stands  as  an  excellent  Canadian 
resource  for  the  introduction  of  effective  new  methods  in  the  lifelong  learning  field.  Widely  copied 
both  within  medicine  (e.g.,  Harvard  Medical  School)  and  without  (all  other  McMaster  faculties),  it 
is  a  respected  source  of  information.  McMaster  holds  a  week-long,  biannual  visitor’s  workshop 
which  serves  as  a  briefing  programme  to  its  methods. 


Conference  on  Interactive  Multimedia  ’96 

This  conference  was  sponsored  by  the  Society  for  Applied  Learning  Technologies  (SALT). 
Section  headings  reflect  particular  themes.  The  following  presentations  are  noteworthy. 


Performance  Support  System  Rescues 

Title:  Use  of  the  Internet  to  Assist  in  Creating  CBT,  Performance  Support  And  Interactive 

Learning  Programs 

Author:  John  Kellum,  VP  and  GM  of  Multimedia  Tools,  Asymetrix  Corp. 

How  to  create  quick,  easy,  cost-effective  CBT,  PSS,  and  interactive  learning  applications. 
Supplied  internal  education  to  Boeing. 


Title:  How  to  Design  and  Build  User-Centred  Integrated  Training  and  Performance 

Support  System 

Authors:  Janis  A.  Morariu,  Peak  Potential,  Inc.  and  Eric  J.  Malecki,  M  Multimedia,  Ltd. 

Describes  their  Integrated  Training  and  Performance  Support  (FTPS)  Design  Methodology,  which 
includes  5  types  of  goals/requirements  and  proceeds  through  five  stages.  Case  study  example  is 
the  FAA’s  Training  and  Evaluation  Support  System  (TESS),  developed  according  to  the 
methodology,  as  the  FAA’s  prototype  model  for  determining  training  cost-benefits  and  Return  on 
Investment  (ROI). 
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Title:  Performance  WebSM  Case  Studies:  How  Training  Organizations  Are  Using  the 

World  Wide  Web  6 

Author:  Michael  Gallagher ,  Director,  Sales  and  Marketing,  AERA,  Inc. 

Application  of  browsers  to  presentation  of  technical  documentation  and  training  via  intranets.  Case 
studies  include  NASA  Goddard. 


Special  Design  Resources 

Tide:  Incorporating  Cognitive  Strategies  and  Metacognition  into  Interactive  CBT 

Author:  Anne-Marie  Armstrong,  Instructional  Designer,  AERA 

Provides  a  guide  to  designing  screens  that  meet  cognitive  learning  objectives,  as  well  as  evaluation 
of  such  objectives. 


Title:  Work  Breakdown  Structures  (WBS):  An  Automated  Systems  Approach  To 

Planning  Technical  Training 

Author:  Darrell  Tatro,  Program  Coordinator,  AEGIS  Training  Center,  and  Marilyn,  B. 

Newman,  Training  Specialist,  AEGIS  Training  Center 

Aid  to  planning  and  implementing  large-scale  training  programs;  can  be  used  as  a  methodology  for 
planning  overall  training  strategy,  and  multimedia  training/tools  within  this  strategy. 


World  Wide  Web  Training  Resources 

Tide:  Internet  Access  for  Beginners  (CD-ROM) 

Authors:  Mary  E.  Sand,  Program  Manager,  Technical  Training,  FAA,  and  Nicholas 

Chandler,  Director  of  Training  Systems,  New  Media  Division,  SAIC 

A  CBI  course  that  teaches  and  uses  the  skills  needed  to  distribute  and  maintain  text-based  and  other 
media-based  teaching/leaming  products  over  the  Internet  (including  CBI  multi-media  materials). 
The  course  teaches  beginning  Internet  skills  so  as  to  avoid  user  frustration,  time  wastage,  etc., 
while  also  serving  as  an  introduction  to  CBT-based  education  and  training. 


Tide:  Training  on  the  World  Wide  Web:  Alternatives  in  Design  and  Development 

Authors:  Ann  Barron,  Associate  Professor,  Instructional  Technology,  University  of  South 

Florida,  Brendan  Tompkins,  Programmer,  and  Kord  Kutchines,  Programmer, 
Analysis  and  Technology,  Inc. 

Presents  new  authoring  tools,  programming  languages  for  interactive  courseware  creation, 
including  HTML,  Java,  Shockwave,  IconAuthor  and  PERL.  Design  guidelines  are  enumerated  as 
well. 
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Tide: 


How  to  Create  Your  Own  Internet-Based  Training  Course 
Author  Pardner  Wynn,  President,  Stanford  Testing  Systems,  Inc. 

Demonstration  of  how  to  create  a  complete  Internet-Based  Training  (IBT)  course  using  IBTauthor, 
an  off-the-shelf  authoring  tool. 


Tide:  US  Department  of  Defense  On-Line  Internet  Journal  for  Training  and  Performance 

Authors:  Leslye  O.  Fuller,  Instructional  Technologist,  Defense  Business  Management 

University,  Bart  Morrison,  Senior  Analyst,  Analysis  and  Technology,  Inc.,  and 
Thomas  J.  Edwards,  Combined  Arms  Support  Command,  US  Army 

On-Line  journal  mounted  to  leverage  the  shared  experience  of  US  forces’  application  of  technology 
to  education  and  training.  Idea  exchange  via  usegroups,  topical  forums,  downloadable  journal 
articles,  demonstrations  of  new  technology,  etc.  Topic  areas  include  human  factors,  cognition, 
CBT,  intelligent  tutors,  simulation,  distance  education,  adult  education,  multimedia,  training 
devices,  etc. 


Tide:  Internet  Treasures  for  Trainers 

Author:  Jodi  Bollaert,  Instructional  Technologist,  Ford  Motor  Company 

Listing  of  web  sites,  gophers,  and  newsgroups  for  training  specialists. 


Tide:  Air  Force  Distance  Training 

Author:  Thomas  E.  Wolfe,  Chief  of  Distance  Learning  Instructional  Design  Branch,  Air 

Force  Distance  Learning  Office 

Covers  the  expansive  growth  of  distance  learning  in  the  USAF  with  CBI,  interactive  courseware, 
and  interactive  multimedia.  Exhaustive  coverage  suitable  to  an  audience  from  beginners  to 
practitioners,  dealing  with  intranets  to  MBONE. 


Multimedia  Development  Resources 

Tide:  Storyboarding  That  Really  Supports  Multimedia  Development 

Authors:  Gary  E.  Davis,  Multimedia  Development  Specialist;  Lucent  Technologies,  and 

Robert  C.  Fratini,  Senior  Advanced  Media  Consultant,  AT&T  Network  Systems; 
Lucent  Technologies 

Lesson  planning  for  case  study  presentation  via  multimedia  can  be  done  easily  by  storyboarding. 
This  paper  describes  the  organisational  factors  needed  to  support  successful  storyboarding,  as  well 
the  Multimedia  Information  Modeler  to  be  used  by  the  SME  in  lesson  planning  for  multimedia. 
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Title:  The  Power  of  Interactive  Video  for  FAA  Course  Development  and  Delivery: 

Lessons  Learned 

Author  Phyllis  Peters  Marson,  FAA 

Paper  on  interactive  video  training  in  the  FAA.  Only  marginally  relevant,  but  included  here  should 
the  interactive  teleconferencing  of  course  delivery  ever  be  considered. 


Title:  Interactive  Multimedia  Learning  Systems 

Author:  Michael  Dulworth,  Insync  Corp. 

A  good  primer  of  multimedia  learning  systems,  including  how  learning  theory  applies,  the 
development  process,  and  authoring  systems.  Includes  a  brief  description  of  the  four  levels  of 
multimedia  learning:  from  customised  linear  presentation,  to  instructor-led/non-linear  presentation, 
to  facilitator-led  training,  to  self-paced  training. 


Title:  Rapid  Scripting  -  A  Non-Traditional  Approach  to  Storyboard  Development 

Author:  Mary  F.  Bratton- Jeffery,  US  Navy 

Current  trends  in  authoring  packages.  Rapid  scripting  was  developed  to  go  beyond  paper  and 
pencil  development  tools  and  produce  synergistic  effects  among  interactive  courseware  production 
team  members. 


Training  and  Development  Resources 


Title: 

Author: 


The  After  Action  Review  Process:  A  Heuristic  Computer  Assisted  Tool  For 
Training 

Robert  C.  Hughes,  Program  Manager,  Cubic  Applications 


After  Action  Review  (AAR)  is  a  computer-assisted  process  designed  to  provide  feedback  after 
computer-assisted  training,  and  to  evaluate  decision-making,  courses  of  actions  selected, 
performance  against  objectives  and  team  interactions.  The  process  promotes  metacognitive 
development  and  awareness  of  the  epistemology  of  what  people  are  actually  doing,  and  is  highly 
compatible  with  the  systems  approach  to  training  and  education  we  have  proposed.  A  meiotic 
teaching  approach,  it  is  based  on  Bloom’s  taxonomy  of  educational  objectives  and  the  metaphor  of 
"candling",  that  is,  observing  without  disturbing  the  group  and  learning  process.  R.  Hughes  is  a 
former  professor  in  the  US  Air  Force  Academy. 


Tide:  Interactive  Multimedia:  Cost  Benefit  Analysis  Issues 

Author:  John  J.  Hirschbuhl,  Assistant  to  the  President,  Information  Services,  The 

University  of  Akron 

Illustration  by  case  studies  of  cost-benefit  analysis  and  positive  ROI  for  networked  multimedia  in 
corporate  environments.  Surprising  results  have  been  achieved.  For  example,  the  US  Institute  for 
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Defense  Analysis,  IBM,  and  American  Airlines  indicate  that  "computer-based  multimedia  produced 
a  savings  of  68%  over  conventional  classroom  training"  (p.  4). 


Innovative  Technologies 

Title:  Automated  Speech  Technologies:  Implications  for  Training  and  Education 

Authors:  Lyn  Gubser,  President,  American  Insights,  Inc.,  and  Thomas  H.  Held,  Director 

Multimedia,  Marketing  and  Product  Development,  Cubic  Applications,  Inc. 

Review  of  work  at  leading  laboratories  in  language  instruction  and  translation,  natural  language 
recognition  systems  and  text-to-speech  (TTS)  systems.  Describes  the  state  of  the  art  in  Automated 
Speech  Technologies  (AST).  This  is  potentially  useful  in  transcription  of  verbal  protocols  in  full 
motion  simulator  sessions  testing  systems  knowledge. 


Title:  On-Line  Dissemination  of  Expert  Knowledge  on  the  Web 

Authors:  Dustin  Huntington,  EXSYS  Inc.,  and  Elias  M.  Awad,  Mclntire  School  of 

Commerce,  University  of  Virginia 

Leading  expert  systems  company  EXSYS  presents  their  approach  to  capturing  and  making 
accessible,  expert  knowledge.  The  system  asks  the  user  focused  relevant  questions  and  comes  to 
conclusions  about  the  state  if  their  knowledge  based  on  the  answers.  Users  are  allowed  to  ask 
questions  in  order  to  expose  the  knowledge  and  inference  structures  of  the  expert  knowledge 
representation.  Provides  explanation  and  recommendations.  Expert  knowledge  is  built  using 
expert  knowledge  development  tools  that  are  files  containing  decision-making  knowledge 
expressed  as  rules  or  facts. 
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